imtoken钱包下载2.0版本|glm
作者： imtoken钱包下载2.0版本
2024-03-07 18:58:20
如何评价清华大学发布的GLM-130B？ - 知乎
如何评价清华大学发布的GLM-130B？ - 知乎首页知乎知学堂发现等你来答切换模式登录/注册自然语言处理清华大学计算机系自然语言生成大规模预训练模型如何评价清华大学发布的GLM-130B？最近清华大学发布了1300亿参数的大模型 GLM-130B，并且开源了代码、模型、API。有人用吗？效果怎么样？显示全部 关注者147被浏览603,354关注问题写回答邀请回答好问题 2添加评论分享14 个回答默认排序OneFlow已认证账号 关注作者｜BBuf、谢子鹏、冯文2017 年，Google 提出了 Transformer 架构，随后 BERT 、GPT、T5等预训练模型不断涌现，并在各项任务中都不断刷新 SOTA 纪录。去年，清华提出了GLM模型（https://github.com/THUDM/GLM），不同于上述预训练模型架构，它采用了一种自回归的空白填充方法， 在 NLP 领域三种主要的任务（自然语言理解、无条件生成、有条件生成）上都取得了不错的结果。很快，清华基于 GLM 架构又推出了 GLM-130B（https://keg.cs.tsinghua.edu.cn/glm-130b/zh/posts/glm-130b/），这是一个开源开放的双语（中文和英文）双向稠密模型，拥有 1300 亿参数，在语言理解、语言建模、翻译、Zero-Shot 等方面都更加出色。预训练模型的背后离不开开源深度学习框架的助力。在此之前，GLM 的开源代码主要是由 PyTorch、DeepSpeed 以及 Apex 来实现，并且基于 DeepSpeed 提供的数据并行和模型并行技术训练了 GLM-Large（335M），GLM-515M（515M），GLM-10B（10B）等大模型，这在一定程度上降低了 GLM 预训练模型的使用门槛。即便如此，对更广大范围的普通用户来说，训练 GLM 这样的模型依然令人头秃，同时，预训练模型的性能优化还有更大的提升空间。为此，我们近期将原始的GLM项目移植到了使用 OneFlow 后端进行训练的 One-GLM 项目。得益于 OneFlow 和 PyTorch 无缝兼容性，我们快速且平滑地移植了 GLM，并成功跑通了预训练任务（训练 GLM-large）。此外，由于 OneFlow 原生支持 DeepSpeed 和 Apex 的很多功能和优化技术，用户不再需要这些插件就可训练 GLM 等大模型。更重要的是，针对当前 OneFlow 移植的 GLM 模型，在简单调优后就能在性能以及显存占用上有大幅提升。具体是怎么做到的？下文将进行揭晓。One-GLM：https://github.com/Oneflow-Inc/one-glmOneFlow：https://github.com/Oneflow-Inc/oneflow1、GLM-large 训练性能和显存的表现首先先展示一下分别使用官方的 GLM 仓库以及 One-GLM 仓库训练 GLM-large 网络的性能和显存表现（数据并行技术），硬件环境为 A100 PCIE 40G，BatchSize 设置为 8。可以看到，在 GLM-large 的训练任务中，相比原始的基于 PyTorch、DeepSpeed、Apex 的 GLM 实现，OneFlow的性能有 120% - 276% 的加速，并且显存占用降低了10% -30%（测试结果均可用 oneflow >=0.9.0 复现）。2、GLM 迁移，只需修改几行代码由于 OneFlow 无缝兼容了 PyTorch 的生态，只需改动几行代码，就可以让用户轻松迁移 GLM 大模型到 One-GLM：将 import torch 替换为 import oneflow as torch将 import torch.xx 替换为 import oneflow.xx将 from apex.optimizers import FusedAdam as Adam 替换为 from oneflow.optim import Adam将 from apex.normalization.fused_layer_norm import FusedLayerNorm as LayerNorm 替换为 from oneflow.nn import LayerNorm注释掉 torch.distributed.ReduceOp，torch.distributed.new_group,，torch.distributed.TCPStore，torch.distributed.all_reduce 这些API，它们是 PyTorch DDP 所需要的，但 OneFlow 的数据并行是由内部的 SBP 和 Global Tensor 机制实现，并不需要这些 API。其它许多模型的迁移更简单，比如在和 torchvision 对标的 flowvision 中，许多模型只需通过在 torchvision 模型文件中加入 import oneflow as torch 即可得到，让用户几乎没有额外成本。此外，OneFlow 还提供全局 “mock torch” 功能（https://docs.oneflow.org/master/cookies/oneflow_torch.html），在命令行运行 eval $(oneflow-mock-torch) 就可以让接下来运行的所有 Python 脚本里的 import torch 都自动指向 oneflow。3、两大调优手段loss 计算部分的优化在原始的 GLM 实现中，loss计算部分使用到了 mpu.vocab_parallel_cross_entropy 这个函数 (https://github.com/THUDM/GLM/blob/main/pretrain_glm.py#L263) 。通过分析这个函数，发现它实现了 sparse_softmax_cross_entropy 的功能，但在实现过程中，原始的 GLM 仓库使用了 PyTorch 的 autograd.Function 模块，并且使用了大量的小算子来拼接出 sparse_softmax_cross_entropy 整体的功能。而在 OneFlow 的算子库中，已经有 sparse_softmax_cross_entropy 这个算子对应的 CUDA 实现了，也就是 flow.sparse_softmax_cross_entropy 这个 API。所以，我们将 GLM 对 sparse_softmax_cross_entropy 的 naive 实现替换为 flow.sparse_softmax_cross_entropy 这个 API，并进行了 loss 对齐实验。结果如何？下图展示了基于 OneFlow 的 Graph 模式训练 GLM-large 模型前 1000 轮的 loss 对齐情况，并分别测试了 FP32 和 AMP 模式：可以看到，将原始 GLM 的 naive sparse_softmax_cross_entropy 实现替换为 flow.sparse_softmax_cross_entropy 之后 loss 是完全对齐的，可以保证正确性。相比原始的 GLM 的单卡性能，这个替换使得 One-GLM 的单卡性能有大幅提升，主要原因是 OneFlow 对 sparse_softmax_cross_entropy 算子做了极致的性能优化，并且减少了原始 GLM 中大量的碎算子拼凑带来的访存开销。此外，这样做也降低了 torch.autograd.Function 本身带来的一些系统开销。CUDA Kernel Fuse除上述优化外，GLM 模型本质上就是一个编解码的 Transformer 架构，所以我们将之前优化 GPT、BERT 的一些 Fuse Pattern 也带给了 One-GLM 模型。具体包含以下两个 Fuse Pattern :fused_bias_add_gelu: 将 bias_add 和 gelu 算子融合在一起。fused_bias_add_dropout：将 bias_add 和 dropout 算子融合在一起。这两个 fuse 都可以显著改善计算的访存，并减少 Kernel Launch 带来的开销，由于 GLM 模型越大则层数就会越多，那么这种 Fuse Pattern 带来的的优势也会不断放大。最终，在上述两方面的优化作用下，在 A100 PCIE 40G，batch_size = 8 环境中的训练 GLM-large 的任务时，单卡 FP32 模式的性能相比原始的 GLM 取得了 280%（FP32 模式） 和 307%（ AMP 模式） 的训练加速。4、LiBai 也能轻松搞定 GLM 推理当模型规模过于庞大，单个 GPU 设备无法容纳大规模模型参数时，便捷好用的分布式训练和推理需求就相继出现，业内也随之推出相应的工具。基于 OneFlow 构建的 LiBai 模型库让分布式上手难度降到最低，用户不需要关注模型如何分配在不同的显卡设备，只需要修改几个配置数据就可以设置不同的分布式策略。当然，加速性能更是出众。LiBai ：https://github.com/Oneflow-Inc/libaiLiBai 相关介绍：大模型训练之难，难于上青天？预训练易用、效率超群的「李白」模型库来了！GLM：https://github.com/Oneflow-Inc/libai/tree/glm_project/projects/GLM用 LiBai 搭建的 GLM 可以便捷地实现model parallel + pipeline parallel推理, 很好地解决单卡放不下大规模模型的问题。那么，用户如何利用大规模模型训练与推理仓库 LiBai 来构建 GLM 的分布式推理部分？下面用一个小例子解释一下。分布式推理具有天然优势要知道，模型的参数其实就是许多 tensor，也就是以矩阵的形式出现，大模型的参数也就是大矩阵，并行策略就是把大矩阵分为多个小矩阵，并分配到不同的显卡或不同的设备上，基础的LinearLayer在LiBai中的实现代码如下：class Linear1D(nn.Module): def __init__(self, in_features, out_features, parallel="data", layer_idx=0, ...): super().__init__() if parallel == "col": weight_sbp = dist.get_nd_sbp([flow.sbp.broadcast, flow.sbp.split(0)]) elif parallel == "row": weight_sbp = dist.get_nd_sbp([flow.sbp.broadcast, flow.sbp.split(1)]) elif parallel == "data": weight_sbp = dist.get_nd_sbp([flow.sbp.broadcast, flow.sbp.broadcast]) else: raise KeyError(f"{parallel} is not supported! Only support ('data', 'row' and 'col')") self.weight = flow.nn.Parameter( flow.empty( (out_features, in_features), dtype=flow.float32, placement=dist.get_layer_placement(layer_idx), # for pipeline parallelism placement sbp=weight_sbp, ) ) init_method(self.weight) ... def forward(self, x): ...在这里，用户可选择去如何切分 Linear 层的矩阵，如何切分数据矩阵，而OneFlow 中的 SBP 控制竖着切、横着切以及其他拆分矩阵的方案（模型并行、数据并行），以及通过设置 Placement 来控制这个 LinearLayer 是放在第几张显卡上（流水并行）。所以，根据 LiBai 中各种 layer 的设计原理以及基于 OneFlow 中 tensor 自带的 SBP 和 Placement 属性的天然优势，使得用户搭建的模型能够很简单地就实现数据并行、模型并行以及流水并行操作。GLM 推理的 Demo 演示这里为用户展示 LiBai 中 GLM 的单卡和便捷的多卡推理 Demo，模型可在 HuggingFace 上获取：https://huggingface.co/models?filter=glm单卡 generate 任务，我们选择 glm-10b 模型：python demo.py
# demo.pyimport oneflow as flowfrom projects.GLM.tokenizer.glm_tokenizer import GLMGPT2Tokenizerfrom libai.utils import distributed as distfrom projects.GLM.configs.glm_inference import cfgfrom projects.GLM.modeling_glm import GLMForConditionalGenerationfrom projects.GLM.utils.glm_loader import GLMLoaderHuggerFacefrom omegaconf import DictConfigtokenizer = GLMGPT2Tokenizer.from_pretrained("/data/home/glm-10b")input_ids = tokenizer.encode( [ "Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai." ], return_tensors="of",)inputs = {"input_ids": input_ids, "attention_mask": flow.ones(input_ids.size())}inputs = tokenizer.build_inputs_for_generation(inputs, max_gen_length=512)sbp = dist.get_nd_sbp([flow.sbp.broadcast, flow.sbp.broadcast])placement = dist.get_layer_placement(0)dist.set_device_type("cpu")loader = GLMLoaderHuggerFace(GLMForConditionalGeneration, cfg, "/path/to/glm-10b")model = loader.load()model = model.half().cuda()dist.set_device_type("cuda")outputs = model.generate( inputs=inputs['input_ids'].to_global(sbp=sbp, placement=placement), position_ids=inputs['position_ids'].to_global(sbp=sbp, placement=placement), generation_attention_mask=inputs['generation_attention_mask'].to_global(sbp=sbp, placement=placement).half(), max_length=512)res = tokenizer.decode(outputs[0])print(res)>>> [CLS] Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.<|endoftext|> <|startofpiece|> Stanford University and a co-founder of <|endofpiece|>
4卡 model parallel+pipeline parallel generate 任务，选择 glm-10b 模型：python3 -m oneflow.distributed.launch --nproc_per_node 4 demo.py
# demo.pyimport oneflow as flowfrom projects.GLM.tokenizer.glm_tokenizer import GLMGPT2Tokenizerfrom libai.utils import distributed as distfrom projects.GLM.configs.glm_inference import cfgfrom projects.GLM.modeling_glm import GLMForConditionalGenerationfrom projects.GLM.utils.glm_loader import GLMLoaderHuggerFacefrom omegaconf import DictConfig# 只需简单配置并行方案parallel_config = DictConfig( dict( data_parallel_size=1, tensor_parallel_size=2, pipeline_parallel_size=2, pipeline_num_layers=2 * 24 ))dist.setup_dist_util(parallel_config)tokenizer = GLMGPT2Tokenizer.from_pretrained("/data/home/glm-10b")input_ids = tokenizer.encode( [ "Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai." ], return_tensors="of",)inputs = {"input_ids": input_ids, "attention_mask": flow.ones(input_ids.size())}inputs = tokenizer.build_inputs_for_generation(inputs, max_gen_length=512)sbp = dist.get_nd_sbp([flow.sbp.broadcast, flow.sbp.broadcast])placement = dist.get_layer_placement(0)loader = GLMLoaderHuggerFace(GLMForConditionalGeneration, cfg, "/path/to/glm-10b")model = loader.load()outputs = model.generate( inputs=inputs['input_ids'].to_global(sbp=sbp, placement=placement), position_ids=inputs['position_ids'].to_global(sbp=sbp, placement=placement), generation_attention_mask=inputs['generation_attention_mask'].to_global(sbp=sbp, placement=placement), max_length=512)res = tokenizer.decode(outputs[0])if dist.is_main_process(): print(res)>>> [CLS] Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.<|endoftext|> <|startofpiece|> Stanford University and a co-founder of <|endofpiece|>
使用 One- GLM 训练的模型进行推理LiBai对于OneFlow的模型加载同样方便，如果你希望使用one-glm训练后的模型进行推理，只需简单的将上述demo中的GLMLoaderHuggerFace替换为GLMLoaderLiBai。5、结语基于 OneFlow 来移植 GLM 大模型非常简单，相比于原始版本 PyTorch GLM 训练 GLM-large 模型，OneFlow 能大幅提升性能和节省显存。此外，通过使用 GLM-10B 这个百亿级大模型做推理，表明基于 OneFlow 的 LiBai 来做大模型推理可以开箱即用，并实现更高的推理速度，如果你想配置不同的并行方式来推理大模型，只需要简单配置文件的几个参数即可。未来，OneFlow团队将探索使用 OneFlow 训练更大的 GLM-130B 千亿模型的可行性，相信基于 OneFlow 可以更快地训练 GLM-130B 千亿级别模型，加速国产大模型训练和推理任务。欢迎Star、试用One-GLM：One-GLM：https://github.com/Oneflow-Inc/one-glmOneFlow：https://github.com/Oneflow-Inc/oneflow其他人都在看35张图，直观理解Stable Diffusion2023年AI十大展望：GPT-4领衔大模型变革李白：你的模型权重很不错，可惜被我没收了OpenAI掌门Sam Altman：AI下一个发展阶段比快更快，开源Stable Diffusion刷新作图速度OneEmbedding:单卡训练TB级推荐模型不是梦“零”代码改动，静态编译让太乙Stable Diffusion推理速度翻倍欢迎Star、试用OneFlow最新版本：发布于 2023-01-20 11:15赞同 493 条评论分享收藏喜欢收起琦琦浙江大学 工学硕士 关注之前笔者已经跟大家详细解析过OpenAI的GPT1～GPT3、InstructGPT、ChatGPT，Anthropic的Claude。随着算力的不断发展，模型容量也越来越大，但这些模型均未开源，走向了Close AI之路。不过即使开源，个体也很难玩转这些模型，计算资源（显卡）、数据集等都是困难。在这样的背景下，国内外涌现出了一批开源模型，近期影响较大的有：Meta AI的LLama、斯坦福基于LLama的Alpaca、清华大学的GLM和ChatGLM等。笔者最近将对这些模型的细节和论文进行详细解析。ChatGLM-6B是一个开源的、支持中英双语问答的对话语言模型，由唐杰团队打造，专门针对中文进行了优化，基于General Language Model (GLM)架构。本文将对GLM结构进行深度剖析和底层原理解读。必备知识在阅读本系列文章之前，建议补充一些相关知识。若你之前未了解过GPT相关原理，可以参考以下的链接：GPT全家桶系列——了解GPT的前世今生琦琦：[万字论文详解]媲美ChatGPT——Google支持的Anthropic推出”更理性的Claude“琦琦：揭开大型语言模型ChatGPT的神秘面纱琦琦：InstructGPT论文精读——ChatGPT前身，从人类反馈中学习琦琦：[万字长文]ChatGPT系列论文精读——大模型经典论文GPT1、GPT2、GPT3琦琦：一文读懂GPT家族和BERT的底层区别——自回归和自编码语言模型详解介绍Gtihub：https://github.com/THUDM/ChatGLM-6B模型文件：https://huggingface.co/THUDM/chatglm-6b博客：https://chatglm.cn/blog论文：https://arxiv.org/pdf/2103.10360.pdf下面进入正题。一、背景前文已经明确阐述了时下主流的预训练框架及其区别。主要有三种：1、autoregressive自回归模型（AR模型）：代表作GPT。本质上是一个left-to-right的语言模型。通常用于生成式任务，在长文本生成方面取得了巨大的成功，比如自然语言生成（NLG）领域的任务：摘要、翻译或抽象问答。当扩展到十亿级别参数时，表现出了少样本学习能力。缺点是单向注意力机制，在NLU任务中，无法完全捕捉上下文的依赖关系。2、autoencoding自编码模型（AE模型）：代表作BERT。是通过某个降噪目标（比如MLM）训练的双向文本编码器。编码器会产出适用于NLU任务的上下文表示，但无法直接用于文本生成。3、encoder-decoder（Seq2seq模型）：代表作T5。采用双向注意力机制，通常用于条件生成任务，比如文本摘要、机器翻译等。三种预训练框架各有利弊，没有一种框架在以下三种领域的表现最佳：自然语言理解（NLU）、无条件生成以及条件生成。T5曾经尝试使用MTL的方式统一上述框架，然而自编码和自回归目标天然存在差异，简单的融合自然无法继承各个框架的优点。在这个天下三分的僵持局面下，GLM诞生了。GLM模型基于autoregressive blank infilling方法，结合了上述三种预训练模型的思想。二、GLM预训练框架GLM有什么特点？又是如何将其他框架的优势巧妙融合的呢？1、自编码思想：在输入文本中，随机删除连续的tokens。2、自回归思想：顺序重建连续tokens。在使用自回归方式预测缺失tokens时，模型既可以访问corrupted文本，又可以访问之前已经被预测的spans。3、span shuffling + 二维位置编码技术。4、通过改变缺失spans的数量和长度，自回归空格填充目标可以为条件生成以及无条件生成任务预训练语言模型。2.1 自回归空格填充任务给定一个输入文本 x=[x_1,...x_n] ，可以采样得到多个文本spans \{s_1,...s_m\} 。为了充分捕捉各spans之间的相互依赖关系，可以对spans的顺序进行随机排列，得到所有可能的排列集合 Z_m ，其中： S_{z GitHub - g-truc/glm: OpenGL Mathematics (GLM) Skip to content Toggle navigation Sign in Product Actions Automate any workflow Packages Host and manage packages Security Find and fix vulnerabilities Codespaces Instant dev environments Copilot Write better code with AI Code review Manage code changes Issues Plan and track work Discussions Collaborate outside of code Explore All features Documentation GitHub Skills Blog Solutions For Enterprise Teams Startups Education By Solution CI/CD & Automation DevOps DevSecOps Resources Learning Pathways White papers, Ebooks, Webinars Customer Stories Partners Open Source GitHub Sponsors Fund open source developers The ReadME Project GitHub community articles Repositories Topics Trending Collections Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name Query To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert g-truc / glm Public forked from icaven/glm Notifications Fork 2k Star 8.5k OpenGL Mathematics (GLM) glm.g-truc.net License View license 8.5k stars 2.1k forks Branches Tags Activity Star Notifications Code Issues 47 Pull requests 5 Actions Projects 0 Security Insights Additional navigation options Code Issues Pull requests Actions Projects Security Insights g-truc/glm This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. masterBranchesTagsGo to fileCode Folders and filesNameNameLast commit messageLast commit dateLatest commit History4,699 Commits.github/workflows.github/workflows cmakecmake docdoc glmglm testtest utilutil .gitignore.gitignore CMakeLists.txtCMakeLists.txt copying.txtcopying.txt manual.mdmanual.md readme.mdreadme.md View all filesRepository files navigationREADMELicense OpenGL Mathematics (GLM) is a header only C++ mathematics library for graphics software based on the OpenGL Shading Language (GLSL) specifications. GLM provides classes and functions designed and implemented with the same naming conventions and functionality than GLSL so that anyone who knows GLSL, can use GLM as well in C++. This project isn't limited to GLSL features. An extension system, based on the GLSL extension conventions, provides extended capabilities: matrix transformations, quaternions, data packing, random numbers, noise, etc... This library works perfectly with OpenGL but it also ensures interoperability with other third party libraries and SDK. It is a good candidate for software rendering (raytracing / rasterisation), image processing, physics simulations and any development context that requires a simple and convenient mathematics library. GLM is written in C++98 but can take advantage of C++11 when supported by the compiler. It is a platform independent library with no dependence and it officially supports the following compilers: GCC 4.7 and higher Intel C++ Compose XE 2013 and higher Clang 3.4 and higher Apple Clang 6.0 and higher Visual C++ 2013 and higher CUDA 9.0 and higher (experimental) Any C++11 compiler For more information about GLM, please have a look at the manual and the API reference documentation. The source code and the documentation are licensed under either the Happy Bunny License (Modified MIT) or the MIT License. Thanks for contributing to the project by submitting pull requests. #include #include #include #include #include #include glm::mat4 camera(float Translate, glm::vec2 const& Rotate) { glm::mat4 Projection = glm::perspective(glm::pi glm::mat4 View = glm::translate(glm::mat4(1.0f), glm::vec3(0.0f, 0.0f, -Translate)); View = glm::rotate(View, Rotate.y, glm::vec3(-1.0f, 0.0f, 0.0f)); View = glm::rotate(View, Rotate.x, glm::vec3(0.0f, 1.0f, 0.0f)); glm::mat4 Model = glm::scale(glm::mat4(1.0f), glm::vec3(0.5f)); return Projection * View * Model; } Latest release Project Health Service Status GitHub actions Build and Install cd /path/to/glm cmake \ -DGLM_BUILD_TESTS=OFF \ -DBUILD_SHARED_LIBS=OFF \ -B build . cmake --build build -- all cmake --build build -- install Passing -DBUILD_SHARED_LIBS=ON to build shared library And then in your CMakeLists.txt: find_package(glm CONFIG REQUIRED) target_link_libraries(main PRIVATE glm::glm) If your prefer to use header-only version of GLM find_package(glm CONFIG REQUIRED) target_link_libraries(main PRIVATE glm::glm-header-only) Vcpkg vcpkg install glm CMake using FetchContent You can add glm to your CMake project to be built together. Add to the CMakeLists.txt file: cmake_minimum_required(VERSION 3.11) # FetchContent is new in version 3.11. include(FetchContent) FetchContent_Declare( glm GIT_REPOSITORY https://github.com/g-truc/glm.git GIT_TAG bf71a834948186f4097caa076cd2663c69a10e1e #refs/tags/1.0.1 ) FetchContent_MakeAvailable(glm) target_link_libraries(main PRIVATE glm::glm) Release notes GLM 1.0.2 - 2024-0X-XX Improvements: Unit tests are not build by default, GLM_BUILD_TESTS set to ON required. GLM 1.0.1 - 2024-02-26 Features: Added C++17 [[nodiscard]] support Improvements: Enables only warnings as errors while building unit tests Added aligned_*vec3 simd support #1245 Fixes: Fixed C++ language auto detection build, disable C++98 warnings with Clang #1235, #1231 Fixed GTX_color_space missing Fixed EXT_matrix_transform shear implementation #1140 #1182 Fixed smoothstep SIMD implementation #1222 GLM 1.0.0 - 2024-01-24 Features: Added GLM_EXT_scalar_reciprocal with tests Added GLM_EXT_vector_reciprocal with tests Added glm::iround and glm::uround to GLM_EXT_scalar_common and GLM_EXT_vector_common Added GLM_EXT_matrix_integer with tests Added Github Actions Added GLM_FORCE_UNRESTRICTED_FLOAT to prevent static asserts when using other scalar types with function expecting floats. Improvements: Added constexpr qualifier for cross product #1040 Added constexpr qualifier for dot product #1040 Fixes: Fixed incorrect assertion for glm::min and glm::max #1009 Fixed quaternion orientation in glm::decompose #1012 Fixed singularity in quaternion to euler angle roll conversion #1019 Fixed quat glm::pow handling of small magnitude quaternions #1022 Fixed glm::fastNormalize build error #1033 Fixed glm::isMultiple build error #1034 Fixed glm::adjugate calculation #1035 Fixed glm::angle discards the sign of result for angles in range (2pi-1, 2pi) #1038 Removed ban on using glm::string_cast with CUDA host code #1041 GLM 0.9.9.8 - 2020-04-13 Features: Added GLM_EXT_vector_intX and GLM_EXT_vector_uintX extensions Added GLM_EXT_matrix_intX and GLM_EXT_matrix_uintX extensions Improvements: Added glm::clamp, glm::repeat, glm::mirrorClamp and glm::mirrorRepeat function to GLM_EXT_scalar_common and GLM_EXT_vector_common extensions with tests Fixes: Fixed unnecessary warnings from matrix_projection.inl #995 Fixed quaternion glm::slerp overload which interpolates with extra spins #996 Fixed for glm::length using arch64 #992 Fixed singularity check for glm::quatLookAt #770 GLM 0.9.9.7 - 2020-01-05 Improvements: Improved Neon support with more functions optimized #950 Added CMake GLM interface #963 Added glm::fma implementation based on std::fma #969 Added missing quat constexpr #955 Added GLM_FORCE_QUAT_DATA_WXYZ to store quat data as w,x,y,z instead of x,y,z,w #983 Fixes: Fixed equal ULP variation when using negative sign #965 Fixed for intersection ray/plane and added related tests #953 Fixed ARM 64bit detection #949 Fixed GLM_EXT_matrix_clip_space warnings #980 Fixed Wimplicit-int-float-conversion warnings with clang 10+ #986 Fixed GLM_EXT_matrix_clip_space perspectiveFov GLM 0.9.9.6 - 2019-09-08 Features: Added Neon support #945 Added SYCL support #914 Added GLM_EXT_scalar_integer extension with power of two and multiple scalar functions Added GLM_EXT_vector_integer extension with power of two and multiple vector functions Improvements: Added Visual C++ 2019 detection Added Visual C++ 2017 15.8 and 15.9 detection Added missing genType check for glm::bitCount and glm::bitfieldReverse #893 Fixes: Fixed for g++6 where -std=c++1z sets __cplusplus to 201500 instead of 201402 #921 Fixed hash hashes qua instead of tquat #919 Fixed .natvis as structs renamed #915 Fixed glm::ldexp and glm::frexp declaration #895 Fixed missing const to quaternion conversion operators #890 Fixed GLM_EXT_scalar_ulp and GLM_EXT_vector_ulp API coding style Fixed quaternion component order: w, {x, y, z} #916 Fixed GLM_HAS_CXX11_STL broken on Clang with Linux #926 Fixed Clang or GCC build due to wrong GLM_HAS_IF_CONSTEXPR definition #907 Fixed CUDA 9 build #910 Deprecation: Removed CMake install and uninstall scripts GLM 0.9.9.5 - 2019-04-01 Fixes: Fixed build errors when defining GLM_ENABLE_EXPERIMENTAL #884 #883 Fixed if constexpr warning #887 Fixed missing declarations for glm::frexp and glm::ldexp #886 GLM 0.9.9.4 - 2019-03-19 Features: Added glm::mix implementation for matrices in *GLM_EXT_matrix_common/ #842 Added CMake BUILD_SHARED_LIBS and BUILD_STATIC_LIBS build options #871 Improvements: Added GLM_FORCE_INTRINSICS to enable SIMD instruction code path. By default, it's disabled allowing constexpr support by default. #865 Optimized inverseTransform #867 Fixes: Fixed in glm::mat4x3 conversion #829 Fixed constexpr issue on GCC #832 #865 Fixed glm::mix implementation to improve GLSL conformance #866 Fixed glm::int8 being defined as unsigned char with some compiler #839 Fixed glm::vec1 include #856 Ignore .vscode #848 GLM 0.9.9.3 - 2018-10-31 Features: Added glm::equal and glm::notEqual overload with max ULPs parameters for scalar numbers #121 Added GLM_FORCE_SILENT_WARNINGS to silent GLM warnings when using language extensions but using W4 or Wpedantic warnings #814 #775 Added adjugate functions to GLM_GTX_matrix_operation #151 Added GLM_FORCE_ALIGNED_GENTYPES to enable aligned types and SIMD instruction are not enabled. This disable constexpr #816 Improvements: Added constant time ULP distance between float #121 Added GLM_FORCE_SILENT_WARNINGS to suppress GLM warnings #822 Fixes: Fixed glm::simplex noise build with double #734 Fixed glm::bitfieldInsert according to GLSL spec #818 Fixed glm::refract for negative 'k' #808 GLM 0.9.9.2 - 2018-09-14 Fixes: Fixed GLM_FORCE_CXX** section in the manual Fixed default initialization with vector and quaternion types using GLM_FORCE_CTOR_INIT #812 GLM 0.9.9.1 - 2018-09-03 Features: Added bitfieldDeinterleave to GLM_GTC_bitfield Added missing glm::equal and glm::notEqual with epsilon for quaternion types to GLM_GTC_quaternion Added GLM_EXT_matrix_relational: glm::equal and glm::notEqual with epsilon for matrix types Added missing aligned matrix types to GLM_GTC_type_aligned Added C++17 detection Added Visual C++ language standard version detection Added PDF manual build from markdown Improvements: Added a section to the manual for contributing to GLM Refactor manual, lists all configuration defines Added missing glm::vec1 based constructors Redesigned constexpr support which excludes both SIMD and constexpr #783 Added detection of Visual C++ 2017 toolsets Added identity functions #765 Split headers into EXT extensions to improve compilation time #670 Added separate performance tests Clarified refract valid range of the indices of refraction, between -1 and 1 inclusively #806 Fixes: Fixed SIMD detection on Clang and GCC Fixed build problems due to std::printf and std::clock_t #778 Fixed int mod Anonymous unions require C++ language extensions Fixed glm::ortho #790 Fixed Visual C++ 2013 warnings in vector relational code #782 Fixed ICC build errors with constexpr #704 Fixed defaulted operator= and constructors #791 Fixed invalid conversion from int scalar with vec4 constructor when using SSE instruction Fixed infinite loop in random functions when using negative radius values using an assert #739 GLM 0.9.9.0 - 2018-05-22 Features: Added RGBM encoding in GLM_GTC_packing #420 Added GLM_GTX_color_encoding extension Added GLM_GTX_vec_swizzle, faster compile time swizzling then swizzle operator #558 Added GLM_GTX_exterior_product with a vec2 glm::cross implementation #621 Added GLM_GTX_matrix_factorisation to factor matrices in various forms #654 Added GLM_ENABLE_EXPERIMENTAL to enable experimental features. Added packing functions for integer vectors #639 Added conan packaging configuration #643 #641 Added glm::quatLookAt to GLM_GTX_quaternion #659 Added glm::fmin, glm::fmax and glm::fclamp to GLM_GTX_extended_min_max #372 Added GLM_EXT_vector_relational: extend glm::equal and glm::notEqual to take an epsilon argument Added GLM_EXT_vector_relational: glm::openBounded and glm::closeBounded Added GLM_EXT_vec1: *vec1 types Added GLM_GTX_texture: levels function Added separate functions to use both negative one and zero near clip plans #680 Added GLM_FORCE_SINGLE_ONLY to use GLM on platforms that don't support double #627 Added GLM_GTX_easing for interpolation functions #761 Improvements: No more default initialization of vector, matrix and quaternion types Added lowp variant of GTC_color_space convertLinearToSRGB #419 Replaced the manual by a markdown version #458 Improved API documentation #668 Optimized GTC_packing implementation Optimized GTC_noise functions Optimized GTC_color_space HSV to RGB conversions Optimised GTX_color_space_YCoCg YCoCgR conversions Optimized GTX_matrix_interpolation axisAngle function Added FAQ 12: Windows headers cause build errors... #557 Removed GCC shadow warnings #595 Added error for including of different versions of GLM #619 Added GLM_FORCE_IGNORE_VERSION to ignore error caused by including different version of GLM #619 Reduced warnings when using very strict compilation flags #646 length() member functions are constexpr #657 Added support of -Weverything with Clang #646 Improved exponential function test coverage Enabled warnings as error with Clang unit tests Conan package is an external repository: https://github.com/bincrafters/conan-glm Clarify quat_cast documentation, applying on pure rotation matrices #759 Fixes: Removed doxygen references to GLM_GTC_half_float which was removed in 0.9.4 Fixed glm::decompose #448 Fixed glm::intersectRayTriangle #6 Fixed dual quaternion != operator #629 Fixed usused variable warning in GLM_GTX_spline #618 Fixed references to GLM_FORCE_RADIANS which was removed #642 Fixed glm::fastInverseSqrt to use fast inverse square #640 Fixed glm::axisAngle NaN #638 Fixed integer pow from GLM_GTX_integer with null exponent #658 Fixed quat normalize build error #656 Fixed Visual C++ 2017.2 warning regarding __has_feature definition #655 Fixed documentation warnings Fixed GLM_HAS_OPENMP when OpenMP is not enabled Fixed Better follow GLSL min and max specification #372 Fixed quaternion constructor from two vectors special cases #469 Fixed glm::to_string on quaternions wrong components order #681 Fixed glm::acsch #698 Fixed glm::isnan on CUDA #727 Deprecation: Requires Visual Studio 2013, GCC 4.7, Clang 3.4, Cuda 7, ICC 2013 or a C++11 compiler Removed GLM_GTX_simd_vec4 extension Removed GLM_GTX_simd_mat4 extension Removed GLM_GTX_simd_quat extension Removed GLM_SWIZZLE, use GLM_FORCE_SWIZZLE instead Removed GLM_MESSAGES, use GLM_FORCE_MESSAGES instead Removed GLM_DEPTH_ZERO_TO_ONE, use GLM_FORCE_DEPTH_ZERO_TO_ONE instead Removed GLM_LEFT_HANDED, use GLM_FORCE_LEFT_HANDED instead Removed GLM_FORCE_NO_CTOR_INIT Removed glm::uninitialize GLM 0.9.8.5 - 2017-08-16 Features: Added Conan package support #647 Fixes: Fixed Clang version detection from source #608 Fixed glm::packF3x9_E1x5 exponent packing #614 Fixed build error min and max specializations with integer #616 Fixed simd_mat4 build error #652 GLM 0.9.8.4 - 2017-01-22 Fixes: Fixed GLM_GTC_packing test failing on GCC x86 due to denorms #212 #577 Fixed POPCNT optimization build in Clang #512 Fixed glm::intersectRayPlane returns true in parallel case #578 Fixed GCC 6.2 compiler warnings #580 Fixed GLM_GTX_matrix_decompose glm::decompose #582 #448 Fixed GCC 4.5 and older build #566 Fixed Visual C++ internal error when declaring a global vec type with siwzzle expression enabled #594 Fixed GLM_FORCE_CXX11 with Clang and libstlc++ which wasn't using C++11 STL features. #604 GLM 0.9.8.3 - 2016-11-12 Improvements: Broader support of GLM_FORCE_UNRESTRICTED_GENTYPE #378 Fixes: Fixed Android build error with C++11 compiler but C++98 STL #284 #564 Fixed GLM_GTX_transform2 shear* functions #403 Fixed interaction between GLM_FORCE_UNRESTRICTED_GENTYPE and glm::ortho function #568 Fixed glm::bitCount with AVX on 32 bit builds #567 Fixed CMake find_package with version specification #572 #573 GLM 0.9.8.2 - 2016-11-01 Improvements: Added Visual C++ 15 detection Added Clang 4.0 detection Added warning messages when using GLM_FORCE_CXX** but the compiler is known to not fully support the requested C++ version #555 Refactored GLM_COMPILER_VC values Made quat, vec, mat type component length() static #565 Fixes: Fixed Visual C++ constexpr build error #555, #556 GLM 0.9.8.1 - 2016-09-25 Improvements: Optimized quaternion glm::log function #554 Fixes: Fixed GCC warning filtering, replaced -pedantic by -Wpedantic Fixed SIMD faceforward bug. #549 Fixed GCC 4.8 with C++11 compilation option #550 Fixed Visual Studio aligned type W4 warning #548 Fixed packing/unpacking function fixed for 5_6_5 and 5_5_5_1 #552 GLM 0.9.8.0 - 2016-09-11 Features: Added right and left handed projection and clip control support #447 #415 #119 Added glm::compNormalize and glm::compScale functions to GLM_GTX_component_wise Added glm::packF3x9_E1x5 and glm::unpackF3x9_E1x5 to GLM_GTC_packing for RGB9E5 #416 Added (un)packHalf to GLM_GTC_packing Added (un)packUnorm and (un)packSnorm to GLM_GTC_packing Added 16bit pack and unpack to GLM_GTC_packing Added 8bit pack and unpack to GLM_GTC_packing Added missing bvec* && and || operators Added glm::iround and glm::uround to GLM_GTC_integer, fast round on positive values Added raw SIMD API Added 'aligned' qualifiers Added GLM_GTC_type_aligned with aligned vec types Added GLM_GTC_functions extension Added quaternion version of glm::isnan and glm::isinf #521 Added glm::lowestBitValue to GLM_GTX_bit #536 Added GLM_FORCE_UNRESTRICTED_GENTYPE allowing non basic genType #543 Improvements: Improved SIMD and swizzle operators interactions with GCC and Clang #474 Improved GLM_GTC_random linearRand documentation Improved GLM_GTC_reciprocal documentation Improved GLM_FORCE_EXPLICIT_CTOR coverage #481 Improved OpenMP support detection for Clang, GCC, ICC and VC Improved GLM_GTX_wrap for SIMD friendliness Added constexpr for *vec*, *mat*, *quat* and *dual_quat* types #493 Added NEON instruction set detection Added MIPS CPUs detection Added PowerPC CPUs detection Use Cuda built-in function for abs function implementation with Cuda compiler Factorized GLM_COMPILER_LLVM and GLM_COMPILER_APPLE_CLANG into GLM_COMPILER_CLANG No more warnings for use of long long Added more information to build messages Fixes: Fixed GLM_GTX_extended_min_max filename typo #386 Fixed glm::intersectRayTriangle to not do any unintentional backface culling Fixed long long warnings when using C++98 on GCC and Clang #482 Fixed sign with signed integer function on non-x86 architecture Fixed strict aliasing warnings #473 Fixed missing glm::vec1 overload to glm::length2 and glm::distance2 functions #431 Fixed GLM test '/fp:fast' and '/Za' command-line options are incompatible Fixed quaterion to mat3 cast function glm::mat3_cast from GLM_GTC_quaternion #542 Fixed GLM_GTX_io for Cuda #547 #546 Deprecation: Removed GLM_FORCE_SIZE_FUNC define Deprecated GLM_GTX_simd_vec4 extension Deprecated GLM_GTX_simd_mat4 extension Deprecated GLM_GTX_simd_quat extension Deprecated GLM_SWIZZLE, use GLM_FORCE_SWIZZLE instead Deprecated GLM_MESSAGES, use GLM_FORCE_MESSAGES instead GLM 0.9.7.6 - 2016-07-16 Improvements: Added pkg-config file #509 Updated list of compiler versions detected Improved C++ 11 STL detection #523 Fixes: Fixed STL for C++11 detection on ICC #510 Fixed missing vec1 overload to length2 and distance2 functions #431 Fixed long long warnings when using C++98 on GCC and Clang #482 Fixed scalar reciprocal functions (GTC_reciprocal) #520 GLM 0.9.7.5 - 2016-05-24 Improvements: Added Visual C++ Clang toolset detection Fixes: Fixed uaddCarry warning #497 Fixed roundPowerOfTwo and floorPowerOfTwo #503 Fixed Visual C++ SIMD instruction set automatic detection in 64 bits Fixed to_string when used with GLM_FORCE_INLINE #506 Fixed GLM_FORCE_INLINE with binary vec4 operators Fixed GTX_extended_min_max filename typo #386 Fixed intersectRayTriangle to not do any unintentional backface culling GLM 0.9.7.4 - 2016-03-19 Fixes: Fixed asinh and atanh warning with C++98 STL #484 Fixed polar coordinates function latitude #485 Fixed outerProduct definitions and operator signatures for mat2x4 and vec4 #475 Fixed eulerAngles precision error, returns NaN #451 Fixed undefined reference errors #489 Fixed missing GLM_PLATFORM_CYGWIN declaration #495 Fixed various undefined reference errors #490 GLM 0.9.7.3 - 2016-02-21 Improvements: Added AVX512 detection Fixes: Fixed CMake policy warning Fixed GCC 6.0 detection #477 Fixed Clang build on Windows #479 Fixed 64 bits constants warnings on GCC #463 GLM 0.9.7.2 - 2016-01-03 Fixes: Fixed GTC_round floorMultiple/ceilMultiple #412 Fixed GTC_packing unpackUnorm3x10_1x2 #414 Fixed GTC_matrix_inverse affineInverse #192 Fixed ICC on Linux build errors #449 Fixed ldexp and frexp compilation errors Fixed "Declaration shadows a field" warning #468 Fixed 'GLM_COMPILER_VC2005 is not defined' warning #468 Fixed various 'X is not defined' warnings #468 Fixed missing unary + operator #435 Fixed Cygwin build errors when using C++11 #405 GLM 0.9.7.1 - 2015-09-07 Improvements: Improved constexpr for constant functions coverage #198 Added to_string for quat and dual_quat in GTX_string_cast #375 Improved overall execution time of unit tests #396 Fixes: Fixed strict alignment warnings #235 #370 Fixed link errors on compilers not supported default function #377 Fixed compilation warnings in vec4 Fixed non-identity quaternions for equal vectors #234 Fixed excessive GTX_fast_trigonometry execution time #396 Fixed Visual Studio 2015 'hides class member' warnings #394 Fixed builtin bitscan never being used #392 Removed unused func_noise.* files #398 GLM 0.9.7.0 - 2015-08-02 Features: Added GTC_color_space: convertLinearToSRGB and convertSRGBToLinear functions Added 'fmod' overload to GTX_common with tests #308 Left handed perspective and lookAt functions #314 Added functions eulerAngleXYZ and extractEulerAngleXYZ #311 Added Added Added static components and precision members to all vector and quat types #350 Added .gitignore #349 Added support of defaulted functions to GLM types, to use them in unions #366 Improvements: Changed usage of __has_include to support Intel compiler #307 Specialized integer implementation of YCoCg-R #310 Don't show status message in 'FindGLM' if 'QUIET' option is set. #317 Added master branch continuous integration service on Linux 64 #332 Clarified manual regarding angle unit in GLM, added FAQ 11 #326 Updated list of compiler versions Fixes: Fixed default precision for quat and dual_quat type #312 Fixed (u)int64 MSB/LSB handling on BE archs #306 Fixed multi-line comment warning in g++. #315 Fixed specifier removal by 'std::make_pair<>' #333 Fixed perspective fovy argument documentation #327 Removed -m64 causing build issues on Linux 32 #331 Fixed isfinite with C++98 compilers #343 Fixed Intel compiler build error on Linux #354 Fixed use of libstdc++ with Clang #351 Fixed quaternion pow #346 Fixed decompose warnings #373 Fixed matrix conversions #371 Deprecation: Removed integer specification for 'mod' in GTC_integer #308 Removed GTX_multiple, replaced by GTC_round GLM 0.9.6.3 - 2015-02-15 Fixed Android doesn't have C++ 11 STL #284 GLM 0.9.6.2 - 2015-02-15 Features: Added display of GLM version with other GLM_MESSAGES Added ARM instruction set detection Improvements: Removed assert for perspective with zFar < zNear #298 Added Visual Studio natvis support for vec1, quat and dualqual types Cleaned up C++11 feature detections Clarify GLM licensing Fixes: Fixed faceforward build #289 Fixed conflict with Xlib #define True 1 #293 Fixed decompose function VS2010 templating issues #294 Fixed mat4x3 = mat2x3 * mat4x2 operator #297 Fixed warnings in F2x11_1x10 packing function in GTC_packing #295 Fixed Visual Studio natvis support for vec4 #288 Fixed GTC_packing packnormx build and added tests #292 Disabled GTX_scalar_multiplication for GCC, failing to build tests #242 Fixed Visual C++ 2015 constexpr errors: Disabled only partial support Fixed functions not inlined with Clang #302 Fixed memory corruption (undefined behaviour) #303 GLM 0.9.6.1 - 2014-12-10 Features: Added GLM_LANG_CXX14_FLAG and GLM_LANG_CXX1Z_FLAG language feature flags Added C++14 detection Improvements: Clean up GLM_MESSAGES compilation log to report only detected capabilities Fixes: Fixed scalar uaddCarry build error with Cuda #276 Fixed C++11 explicit conversion operators detection #282 Fixed missing explicit conversion when using integer log2 with *vec1 types Fixed 64 bits integer GTX_string_cast to_string on VC 32 bit compiler Fixed Android build issue, STL C++11 is not supported by the NDK #284 Fixed unsupported _BitScanForward64 and _BitScanReverse64 in VC10 Fixed Visual C++ 32 bit build #283 Fixed GLM_FORCE_SIZE_FUNC pragma message Fixed C++98 only build Fixed conflict between GTX_compatibility and GTC_quaternion #286 Fixed C++ language restriction using GLM_FORCE_CXX** GLM 0.9.6.0 - 2014-11-30 Features: Exposed template vector and matrix types in 'glm' namespace #239, #244 Added GTX_scalar_multiplication for C++ 11 compiler only #242 Added GTX_range for C++ 11 compiler only #240 Added closestPointOnLine function for tvec2 to GTX_closest_point #238 Added GTC_vec1 extension, *vec1 support to vec types Updated GTX_associated_min_max with vec1 support Added support of precision and integers to linearRand #230 Added Integer types support to GTX_string_cast #249 Added vec3 slerp #237 Added GTX_common with isdenomal #223 Added GLM_FORCE_SIZE_FUNC to replace .length() by .size() #245 Added GLM_FORCE_NO_CTOR_INIT Added 'uninitialize' to explicitly not initialize a GLM type Added GTC_bitfield extension, promoted GTX_bit Added GTC_integer extension, promoted GTX_bit and GTX_integer Added GTC_round extension, promoted GTX_bit Added GLM_FORCE_EXPLICIT_CTOR to require explicit type conversions #269 Added GTX_type_aligned for aligned vector, matrix and quaternion types Improvements: Rely on C++11 to implement isinf and isnan Removed GLM_FORCE_CUDA, Cuda is implicitly detected Separated Apple Clang and LLVM compiler detection Used pragma once Undetected C++ compiler automatically compile with GLM_FORCE_CXX98 and GLM_FORCE_PURE Added not function (from GLSL specification) on VC12 Optimized bitfieldReverse and bitCount functions Optimized findLSB and findMSB functions. Optimized matrix-vector multiple performance with Cuda #257, #258 Reduced integer type redefinitions #233 Rewrote GTX_fast_trigonometry #264 #265 Made types trivially copyable #263 Removed in GLM tests Used std features within GLM without redeclaring Optimized cot function #272 Optimized sign function #272 Added explicit cast from quat to mat3 and mat4 #275 Fixes: Fixed std::nextafter not supported with C++11 on Android #217 Fixed missing value_type for dual quaternion Fixed return type of dual quaternion length Fixed infinite loop in isfinite function with GCC #221 Fixed Visual Studio 14 compiler warnings Fixed implicit conversion from another tvec2 type to another tvec2 #241 Fixed lack of consistency of quat and dualquat constructors Fixed uaddCarray #253 Fixed float comparison warnings #270 Deprecation: Requires Visual Studio 2010, GCC 4.2, Apple Clang 4.0, LLVM 3.0, Cuda 4, ICC 2013 or a C++98 compiler Removed degrees for function parameters Removed GLM_FORCE_RADIANS, active by default Removed VC 2005 / 8 and 2008 / 9 support Removed GCC 3.4 to 4.3 support Removed LLVM GCC support Removed LLVM 2.6 to 3.1 support Removed CUDA 3.0 to 3.2 support GLM 0.9.5.4 - 2014-06-21 Fixed non-utf8 character #196 Added FindGLM install for CMake #189 Fixed GTX_color_space - saturation #195 Fixed glm::isinf and glm::isnan for with Android NDK 9d #191 Fixed builtin GLM_ARCH_SSE4 #204 Optimized Quaternion vector rotation #205 Fixed missing doxygen @endcond tag #211 Fixed instruction set detection with Clang #158 Fixed orientate3 function #207 Fixed lerp when cosTheta is close to 1 in quaternion slerp #210 Added GTX_io for io with #144 Fixed fastDistance ambiguity #215 Fixed tweakedInfinitePerspective #208 and added user-defined epsilon to tweakedInfinitePerspective Fixed std::copy and std::vector with GLM types #214 Fixed strict aliasing issues #212, #152 Fixed std::nextafter not supported with C++11 on Android #213 Fixed corner cases in exp and log functions for quaternions #199 GLM 0.9.5.3 - 2014-04-02 Added instruction set auto detection with Visual C++ using _M_IX86_FP - /arch compiler argument Fixed GTX_raw_data code dependency Fixed GCC instruction set detection Added GLM_GTX_matrix_transform_2d extension (#178, #176) Fixed CUDA issues (#169, #168, #183, #182) Added support for all extensions but GTX_string_cast to CUDA Fixed strict aliasing warnings in GCC 4.8.1 / Android NDK 9c (#152) Fixed missing bitfieldInterleave definisions Fixed usubBorrow (#171) Fixed eulerAngle*** not consistent for right-handed coordinate system (#173) Added full tests for eulerAngle*** functions (#173) Added workaround for a CUDA compiler bug (#186, #185) GLM 0.9.5.2 - 2014-02-08 Fixed initializer list ambiguity (#159, #160) Fixed warnings with the Android NDK 9c Fixed non power of two matrix products Fixed mix function link error Fixed SSE code included in GLM tests on "pure" platforms Fixed undefined reference to fastInverseSqrt (#161) Fixed GLM_FORCE_RADIANS with Fix dot product clamp range for vector angle functions. (#163) Tentative fix for strict aliasing warning in GCC 4.8.1 / Android NDK 9c (#152) Fixed GLM_GTC_constants description brief (#162) GLM 0.9.5.1 - 2014-01-11 Fixed angle and orientedAngle that sometimes return NaN values (#145) Deprecated degrees for function parameters and display a message Added possible static_cast conversion of GLM types (#72) Fixed error 'inverse' is not a member of 'glm' from glm::unProject (#146) Fixed mismatch between some declarations and definitions Fixed inverse link error when using namespace glm; (#147) Optimized matrix inverse and division code (#149) Added intersectRayPlane function (#153) Fixed outerProduct return type (#155) GLM 0.9.5.0 - 2013-12-25 Added forward declarations (glm/fwd.hpp) for faster compilations Added per feature headers Minimized GLM internal dependencies Improved Intel Compiler detection Added bitfieldInterleave and _mm_bit_interleave_si128 functions Added GTX_scalar_relational Added GTX_dual_quaternion Added rotation function to GTX_quaternion (#22) Added precision variation of each type Added quaternion comparison functions Fixed GTX_multiple for negative value Removed GTX_ocl_type extension Fixed post increment and decrement operators Fixed perspective with zNear == 0 (#71) Removed l-value swizzle operators Cleaned up compiler detection code for unsupported compilers Replaced C cast by C++ casts Fixed .length() that should return a int and not a size_t Added GLM_FORCE_SIZE_T_LENGTH and glm::length_t Removed unnecessary conversions Optimized packing and unpacking functions Removed the normalization of the up argument of lookAt function (#114) Added low precision specializations of inversesqrt Fixed ldexp and frexp implementations Increased assert coverage Increased static_assert coverage Replaced GLM traits by STL traits when possible Allowed including individual core feature Increased unit tests completeness Added creating of a quaternion from two vectors Added C++11 initializer lists Fixed umulExtended and imulExtended implementations for vector types (#76) Fixed CUDA coverage for GTC extensions Added GTX_io extension Improved GLM messages enabled when defining GLM_MESSAGES Hidden matrix_inverse function implementation detail into private section GLM 0.9.4.6 - 2013-09-20 Fixed detection to select the last known compiler if newer version #106 Fixed is_int and is_uint code duplication with GCC and C++11 #107 Fixed test suite build while using Clang in C++11 mode Added c++1y mode support in CMake test suite Removed ms extension mode to CMake when no using Visual C++ Added pedantic mode to CMake test suite for Clang and GCC Added use of GCC frontend on Unix for ICC and Visual C++ fronted on Windows for ICC Added compilation errors for unsupported compiler versions Fixed glm::orientation with GLM_FORCE_RADIANS defined #112 Fixed const ref issue on assignment operator taking a scalar parameter #116 Fixed glm::eulerAngleY implementation #117 GLM 0.9.4.5 - 2013-08-12 Fixed CUDA support Fixed inclusion of intrinsics in "pure" mode #92 Fixed language detection on GCC when the C++0x mode isn't enabled #95 Fixed issue #97: register is deprecated in C++11 Fixed issue #96: CUDA issues Added Windows CE detection #92 Added missing value_ptr for quaternions #99 GLM 0.9.4.4 - 2013-05-29 Fixed slerp when costheta is close to 1 #65 Fixed mat4x2 value_type constructor #70 Fixed glm.natvis for Visual C++ 12 #82 Added assert in inversesqrt to detect division by zero #61 Fixed missing swizzle operators #86 Fixed CUDA warnings #86 Fixed GLM natvis for VC11 #82 Fixed GLM_GTX_multiple with negative values #79 Fixed glm::perspective when zNear is zero #71 GLM 0.9.4.3 - 2013-03-20 Detected qualifier for Clang Fixed C++11 mode for GCC, couldn't be enabled without MS extensions Fixed squad, intermediate and exp quaternion functions Fixed GTX_polar_coordinates euclidean function, takes a vec2 instead of a vec3 Clarify the license applying on the manual Added a docx copy of the manual Fixed GLM_GTX_matrix_interpolation Fixed isnan and isinf on Android with Clang Autodetected C++ version using __cplusplus value Fixed mix for bool and bvec* third parameter GLM 0.9.4.2 - 2013-02-14 Fixed compAdd from GTX_component_wise Fixed SIMD support for Intel compiler on Windows Fixed isnan and isinf for CUDA compiler Fixed GLM_FORCE_RADIANS on glm::perspective Fixed GCC warnings Fixed packDouble2x32 on Xcode Fixed mix for vec4 SSE implementation Fixed 0x2013 dash character in comments that cause issue in Windows Japanese mode Fixed documentation warnings Fixed CUDA warnings GLM 0.9.4.1 - 2012-12-22 Improved half support: -0.0 case and implicit conversions Fixed Intel Composer Compiler support on Linux Fixed interaction between quaternion and euler angles Fixed GTC_constants build Fixed GTX_multiple Fixed quat slerp using mix function when cosTheta close to 1 Improved fvec4SIMD and fmat4x4SIMD implementations Fixed assert messages Added slerp and lerp quaternion functions and tests GLM 0.9.4.0 - 2012-11-18 Added Intel Composer Compiler support Promoted GTC_espilon extension Promoted GTC_ulp extension Removed GLM website from the source repository Added GLM_FORCE_RADIANS so that all functions takes radians for arguments Fixed detection of Clang and LLVM GCC on MacOS X Added debugger visualizers for Visual C++ 2012 Requires Visual Studio 2005, GCC 4.2, Clang 2.6, Cuda 3, ICC 2013 or a C++98 compiler GLM 0.9.3.4 - 2012-06-30 Added SSE4 and AVX2 detection. Removed VIRTREV_xstream and the incompatibility generated with GCC Fixed C++11 compiler option for GCC Removed MS language extension option for GCC (not fonctionnal) Fixed bitfieldExtract for vector types Fixed warnings Fixed SSE includes GLM 0.9.3.3 - 2012-05-10 Fixed isinf and isnan Improved compatibility with Intel compiler Added CMake test build options: SIMD, C++11, fast math and MS land ext Fixed SIMD mat4 test on GCC Fixed perspectiveFov implementation Fixed matrixCompMult for none-square matrices Fixed namespace issue on stream operators Fixed various warnings Added VC11 support GLM 0.9.3.2 - 2012-03-15 Fixed doxygen documentation Fixed Clang version detection Fixed simd mat4 /= operator GLM 0.9.3.1 - 2012-01-25 Fixed platform detection Fixed warnings Removed detail code from Doxygen doc GLM 0.9.3.0 - 2012-01-09 Added CPP Check project Fixed conflict with Windows headers Fixed isinf implementation Fixed Boost conflict Fixed warnings GLM 0.9.3.B - 2011-12-12 Added support for Chrone Native Client Added epsilon constant Removed value_size function from vector types Fixed roundEven on GCC Improved API documentation Fixed modf implementation Fixed step function accuracy Fixed outerProduct GLM 0.9.3.A - 2011-11-11 Improved doxygen documentation Added new swizzle operators for C++11 compilers Added new swizzle operators declared as functions Added GLSL 4.20 length for vector and matrix types Promoted GLM_GTC_noise extension: simplex, perlin, periodic noise functions Promoted GLM_GTC_random extension: linear, gaussian and various random number generation distribution Added GLM_GTX_constants: provides useful constants Added extension versioning Removed many unused namespaces Fixed half based type constructors Added GLSL core noise functions GLM 0.9.2.7 - 2011-10-24 Added more swizzling constructors Added missing non-squared matrix products GLM 0.9.2.6 - 2011-10-01 Fixed half based type build on old GCC Fixed /W4 warnings on Visual C++ Fixed some missing l-value swizzle operators GLM 0.9.2.5 - 2011-09-20 Fixed floatBitToXint functions Fixed pack and unpack functions Fixed round functions GLM 0.9.2.4 - 2011-09-03 Fixed extensions bugs GLM 0.9.2.3 - 2011-06-08 Fixed build issues GLM 0.9.2.2 - 2011-06-02 Expend matrix constructors flexibility Improved quaternion implementation Fixed many warnings across platforms and compilers GLM 0.9.2.1 - 2011-05-24 Automatically detect CUDA support Improved compiler detection Fixed errors and warnings in VC with C++ extensions disabled Fixed and tested GLM_GTX_vector_angle Fixed and tested GLM_GTX_rotate_vector GLM 0.9.2.0 - 2011-05-09 Added CUDA support Added CTest test suite Added GLM_GTX_ulp extension Added GLM_GTX_noise extension Added GLM_GTX_matrix_interpolation extension Updated quaternion slerp interpolation GLM 0.9.1.3 - 2011-05-07 Fixed bugs GLM 0.9.1.2 - 2011-04-15 Fixed bugs GLM 0.9.1.1 - 2011-03-17 Fixed bugs GLM 0.9.1.0 - 2011-03-03 Fixed bugs GLM 0.9.1.B - 2011-02-13 Updated API documentation Improved SIMD implementation Fixed Linux build GLM 0.9.0.8 - 2011-02-13 Added quaternion product operator. Clarify that GLM is a header only library. GLM 0.9.1.A - 2011-01-31 Added SIMD support Added new swizzle functions Improved static assert error message with C++0x static_assert New setup system Reduced branching Fixed trunc implementation GLM 0.9.0.7 - 2011-01-30 Added GLSL 4.10 packing functions Added == and != operators for every types. GLM 0.9.0.6 - 2010-12-21 Many matrices bugs fixed GLM 0.9.0.5 - 2010-11-01 Improved Clang support Fixed bugs GLM 0.9.0.4 - 2010-10-04 Added autoexp for GLM Fixed bugs GLM 0.9.0.3 - 2010-08-26 Fixed non-squared matrix operators GLM 0.9.0.2 - 2010-07-08 Added GLM_GTX_int_10_10_10_2 Fixed bugs GLM 0.9.0.1 - 2010-06-21 Fixed extensions errors GLM 0.9.0.0 - 2010-05-25 Objective-C support Fixed warnings Updated documentation GLM 0.9.B.2 - 2010-04-30 Git transition Removed experimental code from releases Fixed bugs GLM 0.9.B.1 - 2010-04-03 Based on GLSL 4.00 specification Added the new core functions Added some implicit conversion support GLM 0.9.A.2 - 2010-02-20 Improved some possible errors messages Improved declarations and definitions match GLM 0.9.A.1 - 2010-02-09 Removed deprecated features Internal redesign GLM 0.8.4.4 final - 2010-01-25 Fixed warnings GLM 0.8.4.3 final - 2009-11-16 Fixed Half float arithmetic Fixed setup defines GLM 0.8.4.2 final - 2009-10-19 Fixed Half float adds GLM 0.8.4.1 final - 2009-10-05 Updated documentation Fixed MacOS X build GLM 0.8.4.0 final - 2009-09-16 Added GCC 4.4 and VC2010 support Added matrix optimizations GLM 0.8.3.5 final - 2009-08-11 Fixed bugs GLM 0.8.3.4 final - 2009-08-10 Updated GLM according GLSL 1.5 spec Fixed bugs GLM 0.8.3.3 final - 2009-06-25 Fixed bugs GLM 0.8.3.2 final - 2009-06-04 Added GLM_GTC_quaternion Added GLM_GTC_type_precision GLM 0.8.3.1 final - 2009-05-21 Fixed old extension system. GLM 0.8.3.0 final - 2009-05-06 Added stable extensions. Added new extension system. GLM 0.8.2.3 final - 2009-04-01 Fixed bugs. GLM 0.8.2.2 final - 2009-02-24 Fixed bugs. GLM 0.8.2.1 final - 2009-02-13 Fixed bugs. GLM 0.8.2 final - 2009-01-21 Fixed bugs. GLM 0.8.1 final - 2008-10-30 Fixed bugs. GLM 0.8.0 final - 2008-10-23 New method to use extension. GLM 0.8.0 beta3 - 2008-10-10 Added CMake support for GLM tests. GLM 0.8.0 beta2 - 2008-10-04 Improved half scalars and vectors support. GLM 0.8.0 beta1 - 2008-09-26 Improved GLSL conformance Added GLSL 1.30 support Improved API documentation GLM 0.7.6 final - 2008-08-08 Improved C++ standard conformance Added Static assert for types checking GLM 0.7.5 final - 2008-07-05 Added build message system with Visual Studio Pedantic build with GCC GLM 0.7.4 final - 2008-06-01 Added external dependencies system. GLM 0.7.3 final - 2008-05-24 Fixed bugs Added new extension group GLM 0.7.2 final - 2008-04-27 Updated documentation Added preprocessor options GLM 0.7.1 final - 2008-03-24 Disabled half on GCC Fixed extensions GLM 0.7.0 final - 2008-03-22 Changed to MIT license Added new documentation GLM 0.6.4 - 2007-12-10 Fixed swizzle operators GLM 0.6.3 - 2007-11-05 Fixed type data accesses Fixed 3DSMax sdk conflict GLM 0.6.2 - 2007-10-08 Fixed extension GLM 0.6.1 - 2007-10-07 Fixed a namespace error Added extensions GLM 0.6.0 : 2007-09-16 Added new extension namespace mecanium Added Automatic compiler detection GLM 0.5.1 - 2007-02-19 Fixed swizzle operators GLM 0.5.0 - 2007-01-06 Upgraded to GLSL 1.2 Added swizzle operators Added setup settings GLM 0.4.1 - 2006-05-22 Added OpenGL examples GLM 0.4.0 - 2006-05-17 Added missing operators to vec* and mat* Added first GLSL 1.2 features Fixed windows.h before glm.h when windows.h required GLM 0.3.2 - 2006-04-21 Fixed texcoord components access. Fixed mat4 and imat4 division operators. GLM 0.3.1 - 2006-03-28 Added GCC 4.0 support under MacOS X. Added GCC 4.0 and 4.1 support under Linux. Added code optimisations. GLM 0.3 - 2006-02-19 Improved GLSL type conversion and construction compliance. Added experimental extensions. Added Doxygen Documentation. Added code optimisations. Fixed bugs. GLM 0.2 - 2005-05-05 Improve adaptative from GLSL. Add experimental extensions based on OpenGL extension process. Fixed bugs. GLM 0.1 - 2005-02-21 Add vec2, vec3, vec4 GLSL types Add ivec2, ivec3, ivec4 GLSL types Add bvec2, bvec3, bvec4 GLSL types Add mat2, mat3, mat4 GLSL types Add almost all functions About OpenGL Mathematics (GLM) glm.g-truc.net Topics opengl cpp vulkan vector matrix mathematics simd quaternion header-only glm cpp-library sycl Resources Readme License View license Activity Custom properties Stars 8.5k stars Watchers 272 watching Forks 2k forks Report repository Releases 38 GLM 1.0.1 Latest Feb 26, 2024 + 37 releases Packages 0 No packages published Languages C++ 96.8% C 2.4% CMake 0.8% Footer © 2024 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time. GLM-130B：开源的双语预训练模型 | GLM-130B GLM-130B GLM-130B：开源的双语预训练模型 八月 4, 2022 | 语言: En 样例演示 ICLR'23 Paper GLM-130B (ICLR'23) 是一个开源开放的双语（中文和英文）双向稠密模型，拥有 1300 亿参数，模型架构采用通用语言模型（GLM1）。它旨在支持在一台 A100（40G * 8） 或 V100（32G * 8）服务器上对千亿规模参数的模型进行推理。截至 2022 年 7 月 3 日，GLM-130B 已完成 4000 亿个文本标识符（中文和英文各 2000 亿）的训练，它有以下独特优势： 双语：同时支持中文和英文。 高精度（英文）： 在 LAMBADA 上优于 GPT-3 175B davinci（+4.0%）、OPT-175B（+5.5%）和 BLOOM-176B（+13.0%），在 MMLU 上略优于 GPT-3 175B（+0.9%）。 高精度（中文）：在 7 个零样本 CLUE 数据集（+24.26%）和 5 个零样本 FewCLUE 数据集（+12.75%）上明显优于 ERNIE TITAN 3.0 260B。 快速推理：支持用一台 A100 服务器使用 SAT 和 FasterTransformer 进行快速推理（提速最高可达 2.5 倍）。 可复现性：所有结果（超过 30 个任务）均可通过我们的开源代码和模型参数轻松复现。 跨平台：支持在 NVIDIA、Hygon DCU、Ascend 910 和 Sunway 处理器上进行训练与推理。 图 1. GLM-130B 的任务表现：与 MMLU 和 LAMBADA 上类似规模的模型相比。(请注意，目前这些均为中间结果，我们尽力做出公平的评估，并欢迎大家一起参与评估。） GLM-130B 模型和推理代码在我们的 GitHub 仓库 公开提供。预训练和微调的代码以及研究论文即将发布。 GLM-130B：构思# 2021 年 12 月，在清华大学知识工程实验室的一次内部头脑风暴会上，我们提出了 GLM-130B 项目的雏形。当时，我们的想法是预训练一个高精度的双语模型（中文/英文），并将其免费开放出来。一来是目前大模型的研究看起来还比较“空中楼阁”，很多研究员都由于缺少算力没法使用，甚至很多企业都用不上；另一方面目前高质量的模型都很少开源开放。GPT-32 是大模型的先驱，但其模型不支持对外开放，而且它也不支持中文。需要注意的是，我们的目标是训练一下1300亿参数（项目代号 “千亿”）的稠密模型，而去年我们研发的“悟道1.75T模型”是一个含 480 个专家混合（MoE）的稀疏模型。我们的理想是世界上任何一个人都可以免费下载千亿模型，并在一台低配的 GPU 服务器上就可以使用它。 然而，我们很快就面临诸多挑战： 缺乏计算资源：很难有机构愿意赞助如此大花费的项目，并免费将其公开。 缺乏高质量的预训练算法：针对双语的高质量预训练算法还有待验证和提升。 缺乏快速推理方法：快速推理方法是保证模型能在低配GPU服务器上运行起来的基础，也是让每个人都能用得上千亿大模型的关键。 对于预训练模型架构算法，我们选择了我们实验室在 21 年提出的 GLM1（ACL'22）模型，其在多个任务上表现出了不俗的性能。然后经过几轮激烈的争论，我们最终决定训练一个 1300 亿参数的 GLM 模型。一来千亿稠密模型能保证高精度，另一方面这个规模还可以在一台 A100 服务器上就进行单机推理。 2022 年 1 月，我们得到了一个 GPU 服务商的小型赞助，开始了我们的第一次测试运行。然而，我们很快发现我们之前大大低估了千亿模型训练的技术难度。预训练一个高精度的千亿模型与训练百亿模型完全不同：频繁的随机硬件故障、模型梯度爆炸、算法中意外的过多内存使用、新的 Megatron 和 DeepSpeed 框架中 3D 流水线的调试、无法从优化器状态中恢复、机器间 TCP 拥塞，以及许多许多意外的 “bug”，项目被多次推迟。清华 PACMAN 团队在这段困难时期向我们伸出了援手，我们一起成功地修复了大部分的 “bug”。 到了 3 月份，我们仍然缺少充足的计算资源，幸运的是我们得到了在其他几个平台上进行测试的机会，包括Ascend 910、Hygon DCU、NVIDIA 和神威。但是直接的难题是我们需要把训练代码适配到这些不同的平台，因为它们的底层算子各不相同（而且很多算子还有所欠缺）。这期间也引入了许多新的技术问题：不支持大维度向量快速计算的 Element-wise 算子，以及阻碍收敛的各种问题——输入嵌入层的过大梯度，Post-LN、Pre-LN3 和Sandwich-LN4 的不稳定性，Dataloader 状态种子恢复，以及 Softmax 和 Attention 的计算精度选择——当然还包括我们自己犯的种种错误。幸运地是，在所有热心合作伙伴的大力帮助下，我们最终能够在所有平台（Ascend 910、Hygon DCU、NVIDIA 和神威）成功运行GLM千亿预训练算法——对我们团队来说，这虽意味着很多个不眠之夜，却也的确是一个意外的收获（跨平台能力）。图 2 中的 GLM-130B 时间轴，涵盖了截至目前我们所遇到和解决的大部分问题。 图 2. 截至2022年7月31日，训练GLM-130B遇到和解决的主要问题的时间轴 4 月 26 日，我们得到了来自 Zhipu.AI（智谱AI——一家旨在“让机器像人一样思考”的人工智能初创公司）慷慨的计算资源赞助。经过又一个星期的测试，我们终于从 5 月 6 日开始在其支持的 96 台 A100（40G*8）服务器上启动了 GLM-130B 模型的训练。此外，智谱还派出了一个工程师团队，协助评估预训练模型，并帮助建设演示网站。 整个训练过程横跨两个月，在此期间，我们开始考虑训练完成后的推理解决方案，并在一台 V100（32G * 8）服务器上实现了合理速度的 130B 模型推理。目前，我们正与清华 NLP 实验室的 BMInf 团队一起探索在一台 RTX-3090 服务器（24G * 8）上使用 GLM-130B 推理的可能性，实现这一目标将可以使更多人用得起千亿模型。 GLM-130B：任务表现# 截至 2022 年 7 月 3 日，GLM-130B 模型已经在超过 4000 亿个文本标识符上进行了训练，其少样本学习的性能在多任务语言理解基准（MMLU）5 上达到并超过了 GPT-3 的水平（参见博客顶部的图 1（左））。GLM-130B 的 5 样本学习性能在训练了 4000 亿个（双语）文本标识符后能达到 44.8% 的准确率。 除了语言理解任务外，我们还在被广泛用于大规模语言模型性能评估的 LAMBADA6 基准上考察了 GLM-130B 的语言建模能力。。图 1（右）展示了相关模型的零样本性能（OPT7 和 BLOOM 的中间结果取自 BLOOM 的评估库）。令人可观的是，GLM-130B 在 LAMBADA（En）上达到了 80.2% 的准确率，而 GPT-3 175B 为 76.2%，此前最好结果为 PaLM 540B的 77.9%8。 由于 GLM-130B 是一个双语（英文和中文）语言模型，我们还在两个中文 NLP 基准上评估它的零样本性能，即 CLUE9 和 FewCLUE10。值得注意的是，中文的下游数据集并不包括在多任务预训练中。GLM-130B 相比目前最大的中文语言模型 ERNIE 3.0 Titan 260B11，在所有数据集上都产生了更好的表现。 图 3. GLM-130B 在部分 CLUE 和 FewCLUE 基准数据集的零样本性能。 最后，我们仍在继续对 GLM-130B 进行广泛的下游任务测试，包括 SuperGLUE12、Big-bench13 等等。我们将随着实验的进行及时与大家分享相应结果。 GLM-130B：模型# 在这一部分，我们将简单介绍一下 GLM-130B 模型背后的技术。更多的细节和源代码可以在项目的 GitHub 仓库 中找到。 GLM-130B 是一个 1300 亿参数规模的双语（中文和英文）双向语言模型。它的底层架构是基于通用语言模型（GLM1），在超过 4000 亿个文本标识符上预训练完成。GLM-130B 利用自回归空白填充作为其主要的预训练目标，以图 4 中的句子为例，它掩盖了随机的连续文本区间（例如，“complete unkown”），并对其进行自回归预测。 图 4. 例子：GLM-130B 在语料“Like a complete unknown, like a rolling stone”进行自回归填空预训练 在实际训练中，GLM-130B 使用两种不同的掩码标识符（[MASK] 和 [gMASK]），分别用于短文和长文的生成。此外，它还采用了最近提出的旋转位置编码（RoPE）14、DeepNorm15 层规范化和高斯误差 GLU（GeGLU）16 17 技术。所有这些设计和技术都对 GLM-130B 大规模语言模型的稳定训练和高精度性能有所帮助。具体来说，GLM-130B 模型含有 70 层 Transformer，隐层维度 12,288，最大序列长度 2,048，以及一个基于 icetk 的 150,000 个标识符的双语分词器。 GLM-130B 对超过 4000 亿个双语标记（2000 亿英文和 2000 亿中文标记）进行了预训练。它的预训练目标由两部分组成：第一部分（95%）是自监督的预训练，即在公开的大规模语料库以及其他一些较小的中文语料库上的自回归空白填充。第二部分（5%）是在 T0++18 和 DeepStruct19 中 70 个不同数据集的抽样子集上进行多任务指令预训练，格式为基于指令的多任务多提示序列到序列的生成。这种设计使 GLM-130B 可以在其他数据集上进行了零样本学习，以及从英文到中文的零样本迁移。 回顾千亿项目的过程，无论从研究、工程、硬件、部署和计算资源的角度，我们都能深刻体会到训练 GLM-130B 其实是一个巨大的挑战，项目中间也面临多次中断的可能。幸运地是，在硬件和模型的测试、训练和评估过程中，GLM-130B 团队尤其是学生负责人——曾奥涵和刘潇同学——付出了巨大的努力，在各种压力下夜以继日地致力于使这个项目存活并最终坚持到取得一定成果。最后，我们向为 GLM-130B 项目提供 GPU 算力支持的赞助商们表示诚挚的谢意。 下一步工作# GLM-130B 的研发仍在继续进行中。与此同时，我们邀请大家加入它的开放社区，推动大规模预训练模型的发展。目前，我们正专注于以下几个方向的研究： GLM-130B 的进一步训练：最近的研究表明，大规模语言模型通常训练不足20。根据Chinchilla 的估计，一个 130B 语言模型的最佳训练标识符训练量应该是 4.0T 左右，比我们目前所训练的数量要大 10 倍。我们正在寻找赞助商和计算平台来支持 GLM-130B 的进一步训练。 INT8 量化：GLM-130B 以 FP16 精度进行训练，总共需要 260G 的 GPU 内存来存储模型权重。DGX-A100 服务器提供了 320G 的 GPU 内存，所以可以天然地支持 GLM-130B。然而，A100 的价格对于绝大多数研究者来说仍然是无法承担的。我们正在对 GLM-130B 模型进行 INT8 量化，以减少推理内存的需求，从而使 GLM-130B 有可能在具有较小 GPU 内存的服务器上运行（例如 8 卡 RTX 3090 GPU）。 混合专家（MoE）方法以扩展模型规模：混合专家模型（Mixture-of-Experts, MoE）已被证明是扩展模型参数的有效方法21 22，然而，MoE模型在相同规模下的表现并不如稠密模型好。我们之前的工作——“悟道1.75T”在基于4.8B的稠密模型，将专家数量扩展到480个以达到1.75万亿的参数。我们正在尝试基于 MoE 技术对 GLM-130B 进行模型扩展，如通过 FastMoE23 及其加速版本 FasterMoE 来进一步扩大它的参数规模，以达到数万亿甚至百万亿规模的参数量，从而获得更高的性能表现。 参数高效 P-Tuning：尽管大规模语言模型具有卓越的零样本和少样本学习能力，通过在下游数据集上对它们进行调整可以进一步提升在特定任务上的性能。然而，它们数量庞大的参数在微调中面临巨大的参数冗余和计算成本。基于我们以前的工作 P-Tuning24 和 P-Tuning v225，我们正在努力尝试将这些技术应用到 GLM-130B 中，以实现参数高效的迁移学习。 致谢 这一项目由国家自然科学基金杰出青年科学基金项目（No. 61825602）支持。 学生负责人# 曾奥涵（清华大学计算机系知识工程实验室），刘潇（清华大学计算机系知识工程实验室） 技术贡献# 清华大学计算机系知识工程实验室 — the Knowledge Engineering Group at Tsinghua# 杜政晓，丁铭，郑勤锴，赖瀚宇，汪子涵，杨卓毅，于济凡，张笑涵，郑问迪，夏箫，徐逸凡，谭咏霖，东昱晓，唐杰 清华大学计算机系 PACMAN 实验室 — the Parallel Architecture & Compiler technology of Mobile, Accelerated, and Networked systems Group at Tsinghua# 马子轩，何家傲，孙桢波，翟季冬，陈文光 清华大学计算机系自然语言处理实验室（BMInf） — the Natural Language Processing Group at Tsinghua# 曾国洋，韩旭，赵威霖，刘知远 智谱AI — an AI startup that aims to teach machines to think like humans# 薛宇飞，王山，陕杰才，姜皓瀚，郭振钢，张鹏 Computation Sponsor# 智谱AI 项目总负责# 唐杰（清华大学计算机系知识工程实验室 & 北京智源人工智能研究院） Du, Zhengxiao, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. “GLM: General Language Model Pretraining with Autoregressive Blank Infilling.” In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 320-335. 2022. ↩︎ ↩︎ ↩︎ Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al. “Language models are few-shot learners.” Advances in neural information processing systems 33 (2020): 1877-1901. ↩︎ Xiong, Ruibin, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. “On layer normalization in the transformer architecture.” In International Conference on Machine Learning, pp. 10524-10533. PMLR, 2020. ↩︎ Ding, Ming, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin et al. “Cogview: Mastering text-to-image generation via transformers.” Advances in Neural Information Processing Systems 34 (2021): 19822-19835. ↩︎ Hendrycks, Dan, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. “Measuring Massive Multitask Language Understanding.” In International Conference on Learning Representations. 2020. ↩︎ Paperno, Denis, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. “The LAMBADA dataset: Word prediction requiring a broad discourse context.” arXiv preprint arXiv:1606.06031 (2016). ↩︎ Zhang, Susan, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan et al. “Opt: Open pre-trained transformer language models.” arXiv preprint arXiv:2205.01068 (2022). ↩︎ Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham et al. “Palm: Scaling language modeling with pathways.” arXiv preprint arXiv:2204.02311 (2022). ↩︎ Xu, Liang, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu et al. “CLUE: A Chinese Language Understanding Evaluation Benchmark.” In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762-4772. 2020. ↩︎ Xu, Liang, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu, Hu Yuan, Guoao Wei et al. “Fewclue: A chinese few-shot learning evaluation benchmark.” arXiv preprint arXiv:2107.07498 (2021). ↩︎ Wang, Shuohuan, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shikun Feng et al. “Ernie 3.0 titan: Exploring larger-scale knowledge enhanced pre-training for language understanding and generation.” arXiv preprint arXiv:2112.12731 (2021). ↩︎ Wang, Alex, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. “Superglue: A stickier benchmark for general-purpose language understanding systems.” Advances in neural information processing systems 32 (2019). ↩︎ Pettee, Mariel, Chase Shimmin, Douglas Duhaime, and Ilya Vidrin. “Beyond imitation: Generative and variational choreography via machine learning.” arXiv preprint arXiv:1907.05297 (2019). ↩︎ Su, Jianlin, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. “Roformer: Enhanced transformer with rotary position embedding.” arXiv preprint arXiv:2104.09864 (2021). ↩︎ Wang, Hongyu, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. “Deepnet: Scaling transformers to 1,000 layers.” arXiv preprint arXiv:2203.00555 (2022). ↩︎ Hendrycks, Dan, and Kevin Gimpel. “Gaussian error linear units (gelus).” arXiv preprint arXiv:1606.08415 (2016). ↩︎ Dauphin, Yann N., Angela Fan, Michael Auli, and David Grangier. “Language modeling with gated convolutional networks.” In International conference on machine learning, pp. 933-941. PMLR, 2017. ↩︎ Sanh, Victor, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin et al. “Multitask Prompted Training Enables Zero-Shot Task Generalization.” In The Tenth International Conference on Learning Representations. 2022. ↩︎ Wang, Chenguang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. “DeepStruct: Pretraining of Language Models for Structure Prediction.” In Findings of the Association for Computational Linguistics: ACL 2022, pp. 803-823. 2022. ↩︎ Hoffmann, Jordan, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas et al. “Training Compute-Optimal Large Language Models.” arXiv preprint arXiv:2203.15556 (2022). ↩︎ Fedus, William, Barret Zoph, and Noam Shazeer. “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” Journal of Machine Learning Research 23, no. 120 (2022): 1-39. ↩︎ Zoph, Barret, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. “Designing effective sparse expert models.” arXiv preprint arXiv:2202.08906 (2022). ↩︎ He, Jiaao, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. “Fastmoe: A fast mixture-of-expert training system.” arXiv preprint arXiv:2103.13262 (2021). ↩︎ Liu, Xiao, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. “GPT understands, too.” arXiv preprint arXiv:2103.10385 (2021). ↩︎ Liu, Xiao, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. “P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.” arXiv preprint arXiv:2110.07602 (2021). ↩︎ © 2022 GLM-130B Powered by Hugo & PaperMod GitHub - THUDM/GLM: GLM (General Language Model) Skip to content Toggle navigation Sign in Product Actions Automate any workflow Packages Host and manage packages Security Find and fix vulnerabilities Codespaces Instant dev environments Copilot Write better code with AI Code review Manage code changes Issues Plan and track work Discussions Collaborate outside of code Explore All features Documentation GitHub Skills Blog Solutions For Enterprise Teams Startups Education By Solution CI/CD & Automation DevOps DevSecOps Resources Learning Pathways White papers, Ebooks, Webinars Customer Stories Partners Open Source GitHub Sponsors Fund open source developers The ReadME Project GitHub community articles Repositories Topics Trending Collections Pricing Search or jump to... Search code, repositories, users, issues, pull requests... Search Clear Search syntax tips Provide feedback We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback Saved searches Use saved searches to filter your results more quickly Name Query To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert THUDM / GLM Public Notifications Fork 304 Star 3k GLM (General Language Model) License MIT license 3k stars 304 forks Branches Tags Activity Star Notifications Code Issues 92 Pull requests 8 Actions Projects 0 Security Insights Additional navigation options Code Issues Pull requests Actions Projects Security Insights THUDM/GLM This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. mainBranchesTagsGo to fileCodeFolders and filesNameNameLast commit messageLast commit dateLatest commit History769 Commits.pytorch_pretrained_bert.pytorch_pretrained_bert chinese_sentencepiecechinese_sentencepiece configconfig config_tasksconfig_tasks data_utilsdata_utils dockerdocker examplesexamples fp16fp16 modelmodel mpumpu scriptsscripts taskstasks testtest .gitignore.gitignore LICENSELICENSE README.mdREADME.md arguments.pyarguments.py blocklm_utils.pyblocklm_utils.py change_mp.pychange_mp.py configure_data.pyconfigure_data.py finetune_glm.pyfinetune_glm.py generate_samples.pygenerate_samples.py generation_utils.pygeneration_utils.py learning_rates.pylearning_rates.py pretrain_glm.pypretrain_glm.py process_grid.pyprocess_grid.py requirements.txtrequirements.txt run_test.pyrun_test.py train_utils.pytrain_utils.py utils.pyutils.py View all filesRepository files navigationREADMEMIT licenseGLM GLM is a General Language Model pretrained with an autoregressive blank-filling objective and can be finetuned on various natural language understanding and generation tasks. Please refer to our paper for a detailed description of GLM: GLM: General Language Model Pretraining with Autoregressive Blank Infilling (ACL 2022) Zhengxiao Du*, Yujie Qian*, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, Jie Tang (*: equal contribution) News: We release ChatGLM-6B, an open pre-trained language model with 6 billion parameters optimized for Chinese QA and dialogue based on the GLM framework. Pretrained Models You can download the pretrained models used in the paper from OneDrive or Tsinghua-Cloud. Name Params Language Corpus Objective File Config GLM-Base 110M English Wiki+Book Token glm-base-blank.tar.bz2 model_blocklm_base.sh GLM-Large 335M English Wiki+Book Token glm-large-blank.tar.bz2 model_blocklm_large.sh GLM-Large-Chinese 335M Chinese WuDaoCorpora Token+Sent+Doc glm-large-chinese.tar.bz2 model_blocklm_large_chinese.sh GLM-Doc 335M English Wiki+Book Token+Doc glm-large-generation.tar.bz2 model_blocklm_large_generation.sh GLM-410M 410M English Wiki+Book Token+Doc glm-1.25-generation.tar.bz2 model_blocklm_1.25_generation.sh GLM-515M 515M English Wiki+Book Token+Doc glm-1.5-generation.tar.bz2 model_blocklm_1.5_generation.sh GLM-RoBERTa 335M English RoBERTa Token glm-roberta-large-blank.tar.bz2 model_blocklm_roberta_large.sh GLM-2B 2B English Pile Token+Sent+Doc glm-2b.tar.bz2 model_blocklm_2B.sh GLM-10B 10B English Pile Token+Sent+Doc Download model_blocklm_10B.sh GLM-10B-Chinese 10B Chinese WuDaoCorpora Token+Sent+Doc Download model_blocklm_10B_chinese.sh Unzip the downloaded file into a local folder and set CHECKPOINT_PATH in the corresponding scripts to the folder path. Results SuperGLUE dev set, single model, single-task finetuning Model COPA WSC RTE WiC CB MultiRC BoolQ ReCoRD GLM-10B 98.0 95.2 93.1 75.7 98.7/98.2 88.1/63.3 88.7 94.4/94.0 DeBERTa-XXLarge-v2 97.0 - 93.5 - - 87.8/63.6 88.3 94.1/93.7 Seq2Seq CNN/Daily Mail (test set, no additional data used) Model ROUGE-1 ROUGE-2 ROUGE-L GLM-10B 44.7 21.4 41.4 T5-11B 43.5 21.6 40.7 PEGASUS-Large 44.2 21.5 41.4 BART-Large 44.2 21.3 40.9 XSum (test set, no additional data used) Model ROUGE-1 ROUGE-2 ROUGE-L GLM-10B 48.9 25.7 40.4 PEGASUS-Large 47.2 24.6 39.3 BART-Large 45.1 22.3 37.3 Language Modeling test set, zero-shot Model LAMBADA (accuracy) Wikitext103 (perplexity) GLM-10B (bi) 72.35 11.33 GLM-10B (uni) 67.18 12.22 GPT-2 52.66 17.48 Megatron-LM (8.3B) 66.51 10.81 Turing-NLG 67.98 10.21 Get Started Hugging Face Hub You can access GLM models via HuggingFace Hub. Please install transformers>=4.23.1 and find all the available models here. Generation from transformers import AutoTokenizer, AutoModelForSeq2SeqLM tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-10b", trust_remote_code=True) model = AutoModelForSeq2SeqLM.from_pretrained("THUDM/glm-10b", trust_remote_code=True) model = model.half().cuda() model.eval() # Inference inputs = tokenizer("Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai.", return_tensors="pt") inputs = tokenizer.build_inputs_for_generation(inputs, max_gen_length=512) inputs = inputs.to('cuda') outputs = model.generate(**inputs, max_length=512, eos_token_id=tokenizer.eop_token_id) print(tokenizer.decode(outputs[0].tolist())) # Training inputs = tokenizer( ["Tsinghua University is located in [MASK].", "One minus one equals zero, is it correct? Answer: [MASK]"], return_tensors="pt", padding=True) inputs = tokenizer.build_inputs_for_generation(inputs, targets=["Beijing", "No"], max_gen_length=8, padding=False) inputs = inputs.to('cuda') outputs = model(**inputs) loss = outputs.loss logits = outputs.logits Classification from transformers import AutoTokenizer, AutoModelForMultipleChoice tokenizer = AutoTokenizer.from_pretrained("THUDM/glm-10b", trust_remote_code=True) model = AutoModelForMultipleChoice.from_pretrained("THUDM/glm-10b", trust_remote_code=True) model = model.half().cuda() model.eval() inputs = tokenizer(["Tsinghua University is located in [MASK].", "One minus one equals zero, is it correct? Answer: [MASK]"], return_tensors="pt", padding=True) choices = [["Beijing", "Shanghai"], ["Yes", "No"]] inputs = tokenizer.build_inputs_for_multiple_choice(inputs, choices) inputs = inputs.to('cuda') outputs = model(**inputs) logits = outputs.logits You can also convert the finetuned checkpoints with scripts/convert_glm_checkpoint_to_transformers.py. Docker Image We prepare two docker images based on CUDA 10.2 and CUDA 11.2. You can pull the pre-built images from Docker Hub and run with docker v19.03+ docker run --gpus all --rm -it --ipc=host zxdu20/glm-cuda102 or replace glm-cuda102 with glm-cuda112. You can also modify the image according to your requirements in docker/cuda102.dockerfile and build the image yourself docker build -f cuda102.dockerfile . -t glm-cuda102 Manual Installation Please first install PyTorch (we use 1.7.0) and apex, and then install other dependencies by pip install -r requirements.txt Clone this repo git clone https://github.com/THUDM/GLM cd GLM Model Parallelism If your encounter the CUDA out of memory error, which means you GPU memory is limited, you can try the model parallelism to divide the parameters into multiple GPUs. Take the two-way model parallelism as an example. First run change_mp.py to divide the checkpoint: python change_mp.py path_to_the_checkpoint 2 Then update the checkpoint path in the model config file (such as config_tasks/model_blocklm_10B.sh) and change MP_SIZE in the script (such as scripts/ds_finetune_superglue.sh) to 2. Usage We provide scripts for finetuning GLM on some downstream tasks. Left-to-Right Generation / Blank Filling (Interactive) Change CHECKPOINT_PATH to your local path. Run the following script bash scripts/generate_block.sh \ config_tasks/model_blocklm_10B_chinese.sh Some models (GLM-2B, GLM-10B, and GLM-10B-Chinese) use three different mask tokens: [MASK] for short blank filling, [sMASK] for sentence filling, and [gMASK] for left-to-right generation. Examples Usage of [MASK] (Entity Prediction): Example1 Context: Ng is an adjunct professor at [MASK] (formerly associate professor and Director of its Stanford AI Lab or SAIL ). Also a pioneer in online education, Ng co-founded Coursera and deeplearning.ai. GLM: the stanford university Example2 (Chinese) Context: 凯旋门位于意大利米兰市古城堡旁。1807年为纪念[MASK]而建，门高25米，顶上矗立两武士青铜古兵车铸像。 GLM:拿破仑军队攻克米兰城 Usage of [sMASK] (Sentence Prediction) Example3 Context: There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and encoder-decoder models (e.g., T5). [sMASK] We propose a General Language Model ( GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over BERT and T5 on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and unconditional generation, GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25× parameters of BERT Large, demonstrating its generalizability to different downstream tasks. GLM: However, there is a growing need to develop a single pretraining model that is not only good at natural language understanding (NLU) or dialog generation/generation (dialog), but is also able to predict other tasks such as sentiment analysis, conditional generation, or machine translation (MT). Example4 (Chinese) Context: 工业互联网（Industrial Internet）是新一代信息通信技术与工业经济深度融合的新型基础设施、应用模式和工业生态，通过对人、机、物、系统等的全面连接，构建起覆盖全产业链、全价值链的全新制造和服务体系，为工业乃至产业数字化、网络化、智能化发展提供了实现途径，是第四次工业革命的重要基石。[sMASK] 它以网络为基础、平台为中枢、数据为要素、安全为保障，既是工业数字化、网络化、智能化转型的基础设施，也是互联网、大数据、人工智能与实体经济深度融合的应用模式，同时也是一种新业态、新产业，将重塑企业形态、供应链和产业链。当前，工业互联网融合应用向国民经济重点行业广泛拓展，形成平台化设计、智能化制造、网络化协同、个性化定制、服务化延伸、数字化管理六大新模式，赋能、赋智、赋值作用不断显现，有力的促进了实体经济提质、增效、降本、绿色、安全发展。 GLM: 工业互联网是制造业技术、管理、模式的重大变革,是推动互联网、大数据、人工智能和实体经济深度融合的重要载体,是建设制造强国和网络强国的重要基础。 Usage of [gMASK] (Long Text Generation) Example5 (Chinese) Context: 问题：冬天，中国哪座城市最适合避寒？问题描述：能推荐一些国内适合冬天避寒的城市吗？回答用户：旅游爱好者 回答： [gMASK] GLM: 海南三亚,三亚的冬天是三亚的旺季,因为冬天国内大部分地方都有雾霾,而三亚却是没有雾霾的地方,所以三亚是冬天最惬意不过的地方了。在东北长大的我觉得三亚简直就是一个天堂,冬天去海口三亚旅游,享受冬天,享受阳光沙滩。但是海口却是很干燥,需要多喝水。 三亚冬天最热门的玩法就是晒太阳,在海边晒晒太阳,在沙滩上晒晒太阳,感觉整个人都得到了解放。三亚还有一个特色项目,就是海上冲浪,在三亚的沙滩上冲浪也是一件非常刺激的事情。 海口,海口冬季的阳光十分温暖,海南的冬季也是属于冬季旅游的旺季。冬季的海口最棒的是去海南的热带野生动植物园,那里有数之不尽的热带小动物,在这里可以近距离的和它们接触,海南的热带野生动植物园也是海南的天然氧吧。还可以在海口观澜湖公园里感受海口美丽的海景。 贵阳,贵州的冬天也是十分温暖的,贵阳也是冬季避寒很好的城市之一。冬季去贵阳玩一定要去黔灵山,黔灵山是贵州香火很旺盛的一个寺庙,寺庙的冬季香火鼎盛,在冬季去寺庙游玩也是一个很好的体验。除了黔灵山,贵阳在冬季还有花溪公园可以去玩,花溪公园也是去当地公园玩最好的选择。 青岛,青岛的冬天是青岛最舒服的时候,青岛有很多海滨浴场,冬天去海边泡一泡温泉,然后晒晒太阳是一件十分惬意的事情。青岛也有沙滩,冬天在沙滩上晒晒太阳,看看海,再玩玩沙滩游戏,感觉十分快乐的事。 You can also add multiple [MASK] and [sMASK] in a single example. The model will fill the blanks one by one from left to right. The answer to each blank always begins with a special <|startofpiece|>. Examples Example1 Context: There have been various types of pretraining architectures including autoencoding models (e.g., BERT), autoregressive models (e.g., GPT), and [MASK] (e.g., T5). [sMASK] We propose a General Language Model ( GLM) based on autoregressive blank infilling to address this challenge. GLM improves blank filling pretraining by adding 2D positional encodings and allowing an arbitrary order to predict spans, which results in performance gains over [MASK] on NLU tasks. Meanwhile, GLM can be pretrained for different types of tasks by varying the number and lengths of blanks. On a wide range of tasks across NLU, conditional and [MASK], GLM outperforms BERT, T5, and GPT given the same model sizes and data, and achieves the best performance from a single pretrained model with 1.25× parameters of BERT Large , demonstrating its generalizability to different downstream tasks. GLM: <|startofpiece|> blank filling models<|startofpiece|> However, most of them cannot easily transfer to other downstream tasks due to the different characteristics of these tasks.<|startofpiece|> other pretrained models<|startofpiece|> unconditional reading, and semantic role labeling tasks Example2 (Chinese) Context: 工业互联网（Industrial Internet）是新一代[MASK]与[MASK]深度融合的新型基础设施、应用模式和工业生态，通过对人、机、物、系统等的全面连接，构建起覆盖全产业链、全价值链的全新制造和服务体系，为工业乃至产业数字化、网络化、智能化发展提供了实现途径，是第四次工业革命的重要基石。[sMASK] 它以网络为基础、平台为中枢、数据为要素、安全为保障，既是工业数字化、网络化、智能化转型的基础设施，也是互联网、大数据、人工智能与实体经济深度融合的应用模式，同时也是一种新业态、新产业，将重塑企业形态、供应链和产业链。当前，工业互联网融合应用向国民经济重点行业广泛拓展，形成[MASK]、智能化制造、[MASK]、个性化定制、服务化延伸、数字化管理六大新模式，赋能、赋智、赋值作用不断显现，有力的促进了实体经济提质、增效、降本、绿色、安全发展。 GLM: <|startofpiece|>信息技术(ICT)<|startofpiece|>工业经济(II2O)<|startofpiece|>我国工业互联网是面向工业全领域、全流程、全体系的互联网,具有多产业、多领域融合的特点。<|startofpiece|>网络化协同<|startofpiece|>平台企业 SuperGLUE Download the SuperGlue data and check the experiment setup in scripts/ds_finetune_superglue.sh. Note that DATA_ROOT, CHECKPOINT_PATH, SAVE_PATH need to be changed to your local path. You may also change the batch-size and nproc_per_node according to your available hardware. Run the following script (use the COPA dataset as an example) bash scripts/ds_finetune_superglue.sh \ config_tasks/model_blocklm_10B.sh \ config_tasks/task_copa.sh We also implement P-Tuning in our code. Run the following script to integrate p-tuning: bash scripts/ds_finetune_superglue_prompt.sh \ config_tasks/model_blocklm_10B.sh \ config_tasks/task_copa.sh To apply GLM to a new NLU dataset with cloze-filling finetuning, implement a DataProcessor in tasks/superglue/dataset.py for data loading and add a PVP in tasks/superglue/pvp.py for the cloze question. More details can be found here. Seq2Seq Download the Gigaword , CNN/Daily Mail or XSum dataset and check the experiment setup in scripts/ds_finetune_seq2seq.sh. Change DATA_ROOT, CHECKPOINT_PATH, SAVE_PATH to your local path. Run the following script (use the CNN/Daily Mail dataset as an example) bash scripts/ds_finetune_seq2seq.sh \ config_tasks/model_blocklm_10B.sh \ config_tasks/seq_cnndm_org.sh The summaries are written into ./runs/experiment_name/test.jsonl.hyps. The references are written into test.jsonl.refs in the same directory. For calculating rouge, install file2rouge and download Stanford CoreNLP from here. Run the following script bash scripts/evaluate_seq2seq.sh \ ./runs/experiment_name/test.jsonl.hyps ./runs/experiment_name/test.jsonl.refs Train with your own data Process your seq2seq data into {split}.source and {split}.target, with each line being the context or the target of a sample, and split being train, val, and test. Run the following script bash scripts/ds_finetune_seq2seq.sh \ config_tasks/model_blocklm_10B.sh \ config_tasks/seq_customization.sh You can specify the hyperparameters in config_tasks/seq_customization.sh and config_tasks/config_blocklm_10B_cnndm.json Multiple Choice (Zero-shot) bash scripts/evaluate_multichoice.sh config_tasks/model_blocklm_10B.sh Note that CHECKPOINT_PATH and DATA_PATH need to be changed to your local path. The format of each line of the data file should be {"inputs_pretokenized": "Context and question here", "choices_pretokenized": ["Choice 1", "Choice 2", "Choice 3"], "label": int} Language Modeling LAMBADA Cloze Accuracy Download the LAMBADA data and change DATA_ROOT, CHECKPOINT_PATH in scripts/evaluate_lm.sh Run the following script bash scripts/evaluate_lm.sh \ config_tasks/model_blocklm_large_generation.sh \ config_tasks/zero_lambada.sh LM Perplexity Download our test set of wikibook or Wikitext103 dataset and change DATA_ROOT, CHECKPOINT_PATH in scripts/evaluate_lm.sh Run the following script bash scripts/evaluate_lm.sh \ config_tasks/model_blocklm_large_generation.sh \ config_tasks/zero_wikitext.sh Text Infilling Download the Yahoo dataset and check the experiment setup in scripts/finetune_blank.sh. Change DATA_ROOT, CHECKPOINT_PATH, SAVE_PATH to your local path. Run the following script bash scripts/finetune_blank.sh \ config_tasks/model_blocklm_large.sh \ config_tasks/seq_blank.sh Pretrain Run the following script to pre-train the GLM-Large model bash scripts/ds_pretrain_nvidia.sh config/ds_block_large.sh The script scripts/ds_pretrain_nvidia.sh launches the training program with DeepSpeed. You should change NUM_WORKERS and NUM_GPUS_PER_WORKER to the number of workers and the number of gpus per worker. Also change HOST_FILE_PATH to the path to an OpenMPI-style hostfile. More details about DeepSpeed launcher can be found here. The file config/ds_block_large.sh defines the hyperparameters for pretraining. Most of the arguments are fairly self-explanatory. Specifically, --train-data can be multiple keywords defined in NAMED_CORPORA in data_utils/corpora.py. The hyperparameters of the optimizer are defined in the corresponding json file under config. The semantics of the json file can be found here. Citation Part of the code is based on Megatron-LM and PET. Please cite our paper if you find this code useful for your research: @article{DBLP:conf/acl/DuQLDQY022, author = {Zhengxiao Du and Yujie Qian and Xiao Liu and Ming Ding and Jiezhong Qiu and Zhilin Yang and Jie Tang}, title = {{GLM:} General Language Model Pretraining with Autoregressive Blank Infilling}, booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), {ACL} 2022, Dublin, Ireland, May 22-27, 2022}, pages = {320--335}, publisher = {Association for Computational Linguistics}, year = {2022}, } About GLM (General Language Model) Resources Readme License MIT license Activity Custom properties Stars 3k stars Watchers 44 watching Forks 304 forks Report repository Releases No releases published Packages 0 No packages published Contributors 15 Languages Python 92.6% Shell 4.9% Dockerfile 2.0% Perl 0.5% Footer © 2024 GitHub, Inc. Footer navigation Terms Privacy Security Status Docs Contact Manage cookies Do not share my personal information You can’t perform that action at this time. GLM（广义线性模型）分析 - 知乎首发于土壤微生物 Soil microbes切换模式写文章登录/注册GLM（广义线性模型）分析傻孩子No one knows everything...简介：在数据分析的过程中，很多分析方法和模型往往要求目标变量（数据）服从某些假设如正态分布、方差齐次等。一般来说，如果数据不能服从这些假设，那么采用对应的方法或模型获得的结果往往不可信。例如，我们经常使用的经典模型，即形如y = kx +b（在R中形如 lm(y ~ x, data)）的一般线性模型就要求数据（目标变量）必须满足正态分布和残差的方差齐次。然而，在实际科研工作中，很多数据往往不能满足以上条件。这种情况就要求我们寻找一种没有以上假设的方法来替代存在假设的模型如：一般线性模型。这种方法之一就是本节我想给大家推荐的广义线性模型（GLM）。广义线性模型，是为了克服线性回归模型的缺点出现的，是线性回归模型的推广。首先自变量可以是离散的，也可以是连续的。离散的可以是0-1变量，也可以是多种取值的变量。广义线性模型取消了对残差(因变量)服从正态分布的要求。残差不一定要服从正态分布，可以服从二项、泊松、负二项、正态、伽马、逆高斯等分布，这些分布被统称为指数分布族。（这一段是我在网上找的，想要进一步了解GLM的，请参考R语言实战或者度娘）在介绍GLM之前，我先说一下为什么我要了解并掌握GLM分析。1）我看到了多篇NC（nature communications）中使用过GLM分析。他们使用GLM要么推断多个自变量对目标变量的解释效应；要么通过算法从很多GLMs中获得最简GLM，然后再根据该GLM预测目标变量的发展趋势；2）看起来这个算法和模型很牛犇。推荐一篇NC供大家在使用该模型时参考“A meta-analysis of global fungal distribution reveals climate-driven patterns.”希望大家看一下我的群公告，在力所能及的情况下帮一下忙，谢谢。1、模型构建在前段时间我介绍的随机森林模型的推文中，使用测试数据，我们发现pH是影响物种丰富度（Richness）的主要因素，其它因素对物种的丰富度均没有显著的影响（如下图）。在计算这个随机森林模型的过程中，我们人为的把pH，CN比、P含量、TC（总碳）、Torigin（初始温度）、ECEC（离子交换量）、CP比、NP比和TN（总氮）作为该模型的一个自变量。最终我们发现这些自变量构成的模型对丰富度的解释量为25.45%。现在问题来了，为什么要选择这些自变量而不是那些自变量作为模型中的一个因子？这些自变量的组合是最优组合吗？这个模型是最优最简模型吗？带着这些问题我们来了解广义线性模型。#load packageslibrary(tidyverse)#install.packages("leaps")library(leaps)#load dataload("RFdata2.RData")head(RFdata2)数据格式如下：1）计算不同GLMs模型对变量的解释效应leaps <- regsubsets(Richness~.,data = RFdata2, nbest=2)plot(leaps, scale = "adjr2")通过全子集回归分析，我们获得了一批模型及其对应的调整R2（如上图）。这个图的左侧纵坐标为调整R2，横坐标为截距和各个自变量，存在颜色表示包含该自变量，空白表示不包含该自变量。我们发现当模型仅有一个变量Torigin时（最下方），GLM模型的调整R2为0.26，而当模型包含Torigin、pH、P、TC、CN_ratio、CP_ratio和NP_ratio时模型的调整R2最大为0.66；相同的当模型包含Torigin、pH、P、TN、CN_ratio、CP_ratio和NP_ratio时模型的调整R2也是最大值0.66。该结果表明这两个模型可能都是解释量最高的模型。为了进一步评估哪个模型是最优模型且同时是最简模型，我们可以看一下每个模型的BIC值，一般来说该值越小则表示模型的拟合度（也就是R2，不是调整R2）越好。plot(leaps, scale = "bic")我们发现不同GLMs的BIC值排序并不与调整R2一致。结果表明了pH、TN、TC和CN_ratio构成的模型以及pH、P、TC和CP_ratio这两个模型的BIC值最低。查看上一个调整R2的值，它们对应的调整R2分别为0.62和0.62。该结果表明这两个模型都是最简模型。因为它们与最大的拟合度0.66只差0.04，因此，从模型的简单性来说，这两个模型就是最优最简模型。根据自己的科研目的可以选择其中之一。最终模型如下：names(RFdata2)fit <- lm(Richness ~ pH+P+TC+CP_ratio, data = RFdata2)summary(fit)通过该结果我们发现，该模型显著影响丰富度，且模型中的每个变量都显著影响丰富度，模型的拟合度为0.66，调整拟合度为0.62。2、模型交叉验证上面我们已经通过算法获得了最优最简模型，那么该模型的稳健性如何呢？下面我们对该模型进行交叉验证。什么叫交叉验证？所谓交叉验证指的是将一定比例的样品挑选出来作为训练样本，另一部分样品作为保留样品，先使用训练样品获得回归方程，然后在保留样品上预测。因为保留样品并没有参与模型的构建过程，因此可以用来估测模型的准确性。k重交叉验证，指的是讲样品分为k个子集，轮流将k-1个子样品作为训练集，另外一个子集作为保留集，最终获得平均预测值。代码如下：#install.packages("bootstrap")library(bootstrap)shrinkage <- function(fit, k = 10){ require(bootstrap) set.seed(123) theta.fit <- function(x,y){lsfit(x,y)} theta.predict <- function(fit,x){cbind(1,x) %*% fit$coef} x <- fit$model[,2:ncol(fit$model)] y <- fit$model[,1] results <- crossval(x, y, theta.fit,theta.predict, ngroup = k) r2 <- cor(y, fit$fitted.values)^2 r2cv <- cor(y, results$cv.fit)^2 cat("Original R-square =", r2, "\n") cat(k, "Fold Cross-Validated R-square =", r2cv, "\n") cat("Change =", r2-r2cv, "\n")}shrinkage(fit, k =10)#Original R-square = 0.6993476 #10 Fold Cross-Validated R-square = 0.527686 #Change = 0.13053710倍交叉验证的结果表明，我们最终获得的模型对丰富度的实际解释量为0.53；变化性为0.13（这相当于误差）。然后通过该模型预测因变量的值如下：#predict valusepredValue <- predict(fit,RFdata2[,c("pH","P","TC","CP_ratio")], interval="predict")predValuefit表示通过该模型预测得到的丰富度值，lwr和upr分别表示下和上误边界。3、模型中每个变量的重要性在获得模型后，我们往往还想要知道获得的模型中每一个变量对自变量如何重要，类似于随机森林分析（可以使用随机森林分析预测）也可以通过以下代码预测（参考R语言实战）。代码和结果如下：#importance of each variablesrelweights <- function(fit,...){ set.seed(123) options(digits = 3) R <- cor(fit$model) nvar <- ncol(R) rxx <- R[2:nvar, 2:nvar] rxy <- R[2:nvar,1] svd <- eigen(rxx) evec <- svd$vectors ev <- svd$values delta <- diag(sqrt(ev)) lambda <- evec %*% delta %*% t(evec) lambdasq <- lambda^2 beta <- solve(lambda) %*% rxy rsquare <- colSums(beta^2) rawwgt <- lambdasq %*% beta^2 import <- (rawwgt/rsquare) *100 import <- as.data.frame(import) rownames(import) <- names(fit$model[2:nvar]) names(import) <- "Weights" dotchart(import$Weights, labels = rownames(import), xlab = "% of R-Square", pch = 19, main = "Relative importance of predictor variables", sub = paste("Total R-Square =",round(rsquare,digits = 2)), ...) return(import)}relweights(fit,col = "blue")跟我们的随机森林分析的结果对照，GLM模型的结果表明了pH是影响richness的最主要影响因素。其次是CP比，影响最小的是TC。分析中的数据可以添加我们的微信群获得，获得途径1，关注本微信公众号（科白君的土壤世界），后台回复 “客服微信”，小编将邀请您进群和我们一起交流和学习~发布于 2021-10-27 12:07广义线性模型计量经济计量经济学赞同 285 条评论分享喜欢收藏申请转载文章被以下专栏收录土壤微生物 Soil microbes微生物知识和数 广义线性模型（GLM）和广义线性混合模型（GLMM）怎么区分使用呢？ - 知乎首页知乎知学堂发现等你来答切换模式登录/注册统计学回归分析广义线性模型广义线性模型（GLM）和广义线性混合模型（GLMM）怎么区分使用呢？什么情况下可以用GLM，什么情况下必须用GLMM呢关注者191被浏览237,867关注问题写回答邀请回答好问题 10添加评论分享8 个回答默认排序和煦星光生物学话题下的优秀答主 关注说下自己的理解，权当抛砖引玉。首先，题主问题有误，GLM一般是指 generalized linear model ，也就是广义线性模型；而非 general linear model，也就是一般线性模型；而GLMM （generalized linear mixed model）是广义线性混合模型。广义线性模型GLM很简单，举个例子，药物的疗效和服用药物的剂量有关。这个相关性可能是多种多样的，可能是简单线性关系（发烧时吃一片药退烧0.1度，两片药退烧0.2度，以此类推；这种情况就是一般线性模型），也可能是比较复杂的其他关系，如指数关系（一片药退烧0.1度，两片药退烧0.4度），对数关系等等。这些复杂的关系一般都可以通过一系列数学变换变成线性关系，以此统称为广义线性模型。广义线性混合模型GLMM比较复杂，GLM要求观测值误差是随机的，而GLMM则要求误差值并非随机，而是呈一定分布的。举个例子，我们认为疗效可能与服药时间相关，但是这个相关并不是简简单单的疗效随着服药时间的变化而改变。更可能的是疗效的随机波动的程度与服药时间有关。比如说，在早上10：00的时候，所有人基本上都处于半饱状态，此时吃药，相同剂量药物效果都差不多。但在中午的时候，有的人还没吃饭， 有的人吃过饭了，有的人喝了酒，结果酒精和药物起了反应，有的人喝了醋，醋又和药物起了另一种反应。显然，中午吃药会导致药物疗效的随机误差非常大。这种疗效的随机误差（而非疗效本身）随着时间的变化而变化，并呈一定分布的情况，必须用广义线性混合模型了。编辑于 2015-02-02 22:25赞同 16512 条评论分享收藏喜欢收起极速BigStone 关注为了说明的方便，默认GLM包含了“经典线性回归模型”，GLMM包含了“线性混合模型”。在只有GLM和GLMM两种选择的情况下，响应变量y如果是独立的则GLM（或者GLMM，但是不推荐，因为GLMM回归系数估计算法复杂并且不会比GLM算法更好），如果不独立，相关，必须GLMM。什么情况下不独立呢？比如重复测量/纵向数据，面板数据，聚集数据等。举个例子，研究学生成绩的影响因素。学生来自不同学校和不同班级，很显然，同一个班级的学生成绩是相关的，同一个学校的学生成绩可能是相关的。像这样的数据你使用GLM的结果就是回归系数的标准误会被严重低估，造成回归系数容易变得显著。另外学校总体或者班级总体的差异你也得不到。所以判断标准非常简单，响应变量y独立则GLM，否则GLMM。这就是最本质的判断标准，其它的标准都是基于这个标准来的。比如有的说有随机效应就要GLMM，为什么会有随机效应，还不是因为不独立，增加随机效应后就能体现y之间的相关性。指出最高赞的一个错误：广义线性混合模型GLMM比较复杂，GLM要求观测值误差是随机的，而GLMM则要求误差值并非随机，而是呈一定分布的。误差是随机变量，随机变量还能不随机吗？！整段话不知所云。我猜想他应该是没有用正确的语言清楚的表达出他实际要说的话。我此处指出这个错误无恶意，毕竟是最高赞回答，我不能看到错误装作看不见，这对于广大求知者是不负责的。接着指出其它回答中的错误。有回答说GLM要求“方差齐性”，不需要的，因为二项分布，泊松分布，负二项分布等都是异方差，GLM没有“方差齐性”这个要求。GLM“残差的正态性（Normality of residuals）”也是不需要的。编辑于 2021-02-14 11:40赞同 475 条评论分享收藏喜欢 广义线性模型（GLM）及其应用 - 知乎首发于deephub深度学习切换模式写文章登录/注册广义线性模型（GLM）及其应用deephubAI方向文章，看头像就知道，这里都是"干"货广义线性模型[generalize linear model(GLM)]是线性模型的扩展，通过联系函数建立响应变量的数学期望值与线性组合的预测变量之间的关系。它的特点是不强行改变数据的自然度量，数据可以具有非线性和非恒定方差结构。是线性模型在研究响应值的非正态分布以及非线性模型简洁直接的线性转化时的一种发展。在广义线性模型的理论框架中，则假设目标变量Y则是服从指数分布族，正态分布和伯努利分布都属于指数分布族，因此线性回归和逻辑回归可以看作是广义线性模型的特例。这是概率分布及其正则联系函数（Canonical Link function）的列表。正态分布:恒等函数泊松分布:对数函数二项分布:分对数函数 除此以外我们还可以自定义联系函数，如果不喜欢自己编写可以使用在 statsmodels 中实现了的各种联系函数，Stan、PyMC3 和 TensorFlow Probability 等概率编程框架也给我们提供了这些函数。link function也被翻译为连接函数，这里觉得联系函数更为贴切所以还是翻译为联系函数线性回归线性回归用于通过解释变量 X 的线性组合来预测连续变量 y 的值。在单变量情况下，线性回归可以表示如下模型假定噪声项的正态分布。 该模型可以说明如下泊松回归泊松分布用于对计数数据进行建模。 它只有一个参数代表分布的均值和标准差。 这意味着平均值越大，标准差越大。如果我们将泊松回归应用于数据。 结果应该是这样的。预测曲线是指数的，因为对数联系函数（ log link function）的反函数是指数函数。 由此也可以清楚地看出，由线性预测器计算的泊松回归参数保证为正。以下是一个泊松回归的示例代码import numpy as np from numpy.random import uniform, normal, poisson, binomial from scipy import stats import matplotlib.pyplot as plt import seaborn as sns import statsmodels.api as sm %matplotlib inline ## ============Poisson regression # generate simulation data np.random.seed(5) n_sample = 100 a = 0.6 b = -0.4 x = uniform(1, 5, size=n_sample) mu = np.exp(a * x + b) y = poisson(mu) import statsmodels.api as sm exog, endog = sm.add_constant(x), y # Poisson regression mod = sm.GLM(endog, exog, family=sm.families.Poisson(link=sm.families.links.log())) res = mod.fit() display(res.summary()) y_pred = res.predict(exog) idx = x.argsort() x_ord, y_pred_ord = x[idx], y_pred[idx] plt.plot(x_ord, y_pred_ord, color='m') plt.scatter(x, y, s=20, alpha=0.8) plt.xlabel("X") plt.ylabel("Y")粉色曲线是泊松回归的预测。逻辑回归如果使用分对数（ logit）函数作为联系函数，使用二项式/伯努利分布作为概率分布，则该模型称为逻辑回归。第二个方程的右边叫做logistic函数。因此这个模型被称为逻辑回归。对于任意输入，logistic函数返回的值在0到1之间，对于二项分布它是一个合适的联系函数。逻辑回归也就是我们常看到的这个样子总结如果要进行“广义线性模型（GLM）”分析，只需要摘到我们需要的联系函数，它的作用就是把Y与X间的非线性关系转换成线性关系，我们完全可以自己编写我们需要的联系函数。实际使用中我们只要把联系函数和方差函数假设正确，甚至不用管是什么分布的，如果使用的就是一些典型联系函数，则方差函数都可以不用假设。所以其实广义线性模型的要点就是：联系函数和/或方差函数要假设正确，这样就ok了。本文代码https://github.com/ranasingh-gkp/StatisticalModeling_Implement/blob/main/GLM.ipynb作者：Rana singh发布于 2022-08-20 10:04广义线性模型机器学习Python赞同 9添加评论分享喜欢收藏申请转载文章被以下专栏收录deephub深度学习提供专业的CV NLP和数据挖 广义线性模型（GLM）概述 - 知乎首发于统计模型切换模式写文章登录/注册广义线性模型（GLM）概述休絮好乐无荒！22.01.08期末考完修改了一些笔误某节大数据专业必修课的专题之一，结果发现比自己的统计学专业课还硬核（又硬又核）。广义线性模型（Generalized Linear Model）是线性回归模型的推广，在很多领域都有广泛应用，比如逻辑斯谛回归（logistic regression model）就是响应变量服从两点分布并取连接函数为logit函数时的特例，据说生存分析也有不少相关应用，但目前还没学到2333。某乎上介绍相关理论的内容很少，初学的时候整个人都处于一种升华状态，于是打算整理一下过去几节课的笔记。主要是介绍了广义线性模型的基本构架，然后讲了一下用MLE对模型作估计以及估计的一些性质。还加了一些自己的推导。latex误事，一个周末交代在这个文件上了，大三还是得好好学习，少碰某乎啊……以后随缘更新吧23333附：pdf文件GLM.pdf216.6K · 百度网盘编辑于 2022-01-08 12:36广义线性模型赞同 18117 条评论分享喜欢收藏申请转载文章被以下专栏收录统计模型鬼知道我能学到些啥，整天看些自己够不着 怎样简单易懂的解释GLM（广义线性模型）？ - 知乎首页知乎知学堂发现等你来答切换模式登录/注册数学统计学非线性R（编程语言）怎样简单易懂的解释GLM（广义线性模型）？如题。敬请大神不吝赐教。 望能从线性模型的推广上讲起。显示全部 关注者26被浏览25,658关注问题写回答邀请回答好问题 12 条评论分享5 个回答默认排序斯宾王基金从业资格证持证人 关注广义线性模型（GLM）假设条件概率分布 y|x （ y\in\mathbb{R} 是一个被预测变量数据点， x\in\mathbb{R}^p 是一个预测变量数据点）属于指数分布族f_\theta(y)=\exp(\theta y-c(\theta))h(y)\\这里 \theta\in\mathbb{R} 是概率分布参数， c:\mathbb{R}\to\mathbb{R},h:\mathbb{R}\to\mathbb{R}_+ 是已知函数（ h 是非负函数）。指数分布族包括正态分布、泊松分布、伯努利分布和多项分布等。指数分布族满足性质 \mathbb{E}[y|x]=c'(\theta),\mathrm{Var}(y|x)=c''(\theta) ，而GLM的联系函数 g:\mathbb{R}\to\mathbb{R} 用来将条件期望转为 x 的线性函数g(\mathbb{E}[y|x])=\beta^\top x\\标准线性回归模型是GLM的一种，它假设 y 给定 x 的条件概率分布是正态分布 y|x\sim N(\mathbb{E}[y|x],\sigma^2) ，这里联系函数是恒等函数 g=\mathrm{id} ，即 \mathbb{E}[y|x]=\beta^\top x 。当被预测变量是非负整数时，我们可以使用柏松回归；它假设 y 给定 x 的条件概率分布是柏松分布 y|x\sim\mathrm{Poisson}(\mathbb{E}[y|x]) ，这里联系函数是 g=\log ，即\log(\mathbb{E}[y|x])=\beta^\top x\\且 \mathrm{Var}(y|x)=\mathbb{E}[y|x] 。若被预测变量是二元变量，我们可以使用逻辑回归；它假设 y 给定 x 的条件概率分布是伯努利分布 y|x\sim\mathrm{Ber}(\mathbb{E}[y|x]) ，这里联系函数是 g=\mathrm{logit},g^{-1}=\mathrm{sigmoid} ，即\log(\frac{\mathbb{E}[y|x]}{1-\mathbb{E}[y|x]})=\beta^\top x\\ \frac{\exp(\beta^\top x)}{1+\exp(\beta^\top x)}=\mathbb{E}[y|x]\\且 \mathrm{Var}(y|x)=\mathbb{E}[y|x](1-\mathbb{E}[y|x]) 。若被预测变量是分类变量，我们可以使用多项回归；假设 y\in\{1,\cdots,K\} ，令 \mathbb{P}(y=k|x)=\mathbb{E}_k[y|x] ，我们有\log(\frac{\mathbb{E}_k[y|x]}{\mathbb{E}_K[y|x]})=\beta_k^\top x,k=1,\cdots,K-1\\ \mathbb{E}_k[y|x]=\frac{\exp(\beta_k^\top x)}{1+\sum_{l=1}^{K-1}\exp(\beta_l^\top x)},k=1,\cdots,K-1\\ \mathbb{E}_K[y|x]=\frac{1}{1+\sum_{l=1}^{K-1}\exp(\beta_l^\top x)}\\和M估计一样，GLM系数通常没有闭式表达式，故GLM系数经常通过最大似然估计（MLE）来计算\nabla l(\beta)=0,l(\beta)=\sum_{i=1}^n\log(f_{\theta_i}(y_i)),\theta_i=\beta^\top X_i\\这里 l(\beta) 叫作对数似然函数， \nabla l(\beta) 叫作得分向量。我们可以使用牛顿法\beta\leftarrow\beta+J(\beta)^{-1}\nabla l(\beta)\\来解此求根问题，这里 J(\beta)=\nabla^2l(\beta) 是被观察到的信息矩阵。若 J(\beta) 被费希尔信息矩阵 I(\beta)=\mathbb{E}[\nabla l(\beta)\nabla l(\beta)^\top] 替代，此迭代法就被称作迭代再加权最小二乘法（IRLS）。编辑于 2023-03-29 12:01赞同 11添加评论分享收藏喜欢收起学习者老张AI已来 关注GLMs are a type of statistical model that extends the linearregression model to accommodate non-normal distributionsand non-constant variance. In other words, GLMs allow youto model data that is not normally distributed or has unequalvariance.GLMs consist of three components: a random component, asystematic component, and a link function. The randomcomponent specifies the probability distribution of theresponse variable, the systematic component relates theresponse variable to the predictor variables, and the linkfunction connects the two components.For example, if you have binary data (0 or 1), you can use aBernoulli distribution as the random component, a logisticfunction as the link function, and a linear combination ofpredictor variables as the systematic component. This wouldgive you a logistic regression model, which can be used topredict the probability of a binary outcome based on thepredictor variables. hope this explanation helps you understand GLMs better!Let me know if you have any further questions.发布于 2023-03-29 09:24赞同添加评论分享收藏喜欢收起
GLM-130B：开源的双语预训练模型 | GLM-130B
GitHub - THUDM/GLM: GLM (General Language Model)
GLM（广义线性模型）分析 - 知乎
广义线性模型（GLM）和广义线性混合模型（GLMM）怎么区分使用呢？ - 知乎
广义线性模型（GLM）及其应用 - 知乎
广义线性模型（GLM）概述 - 知乎
怎样简单易懂的解释GLM（广义线性模型）？ - 知乎