当前位置： 首页 > news >正文

无锡网站推广优化费用精仿源码社区网站源码

news 2025/9/17 17:02:48
无锡网站推广优化费用,精仿源码社区网站源码,成立公司需要什么条件,表格在网站后台是居中可到前台为什么不居中打卡 目录 打卡 环境准备 准备阶段 数据加载与预处理 BertTokenizer 部分输出 模型构建 gpt2模型结构输出 训练流程 部分输出 部分输出2（减少训练数据） 推理流程 环境准备 pip install -i https://pypi.mirrors.ustc.edu.cn/simple mindspo…

打卡

目录

打卡

环境准备

准备阶段

数据加载与预处理

BertTokenizer

部分输出

模型构建

gpt2模型结构输出

训练流程

部分输出

部分输出2（减少训练数据）

推理流程

环境准备

pip install -i https://pypi.mirrors.ustc.edu.cn/simple mindspore==2.2.14pip install tokenizers==0.15.0 -i https://pypi.tuna.tsinghua.edu.cn/simple
# 该案例在 mindnlp 0.3.1 版本完成适配，如果发现案例跑不通，可以指定mindnlp版本，执行`!pip install mindnlp==0.3.1`pip install mindnlp

准备阶段

nlpcc2017摘要数据，内容为新闻正文及其摘要，总计50000个样本。

来源：nlpcc2017摘要数据

数据加载与预处理

  • 原始数据格式：
article: [CLS] article_context [SEP]
summary: [CLS] summary_context [SEP]
  • 预处理后的数据格式：
[CLS] article_context [SEP] summary_context [SEP]

BertTokenizer

因GPT2无中文的tokenizer，使用BertTokenizer替代。代码如下：

from mindspore.dataset import TextFileDataset
import json
import numpy as np
from mindnlp.transformers import BertTokenizer# preprocess dataset
def process_dataset(dataset, tokenizer, batch_size=6, max_seq_len=1024, shuffle=False):def read_map(text):data = json.loads(text.tobytes())return np.array(data['article']), np.array(data['summarization'])def merge_and_pad(article, summary):# tokenization# pad to max_seq_length, only truncate the articletokenized = tokenizer(text=article, text_pair=summary,padding='max_length', truncation='only_first', max_length=max_seq_len)return tokenized['input_ids'], tokenized['input_ids']dataset = dataset.map(read_map, 'text', ['article', 'summary'])# change column names to input_ids and labels for the following trainingdataset = dataset.map(merge_and_pad, ['article', 'summary'], ['input_ids', 'labels'])dataset = dataset.batch(batch_size)if shuffle:dataset = dataset.shuffle(batch_size)return dataset# load dataset
dataset = TextFileDataset(str(path), shuffle=False)
print(dataset.get_dataset_size())   ### 50000# split into training and testing dataset
train_dataset, test_dataset = dataset.split([0.9, 0.1], randomize=False)
print(len(train_dataset))  ### 45000# We use BertTokenizer for tokenizing chinese context.
tokenizer = BertTokenizer.from_pretrained('bert-base-chinese')
len(tokenizer)train_dataset = process_dataset(train_dataset, tokenizer, batch_size=4)
## next(train_dataset.create_tuple_iterator())

部分输出

模型构建

如下，通过两个类实现：

  1. 构建GPT2ForSummarization模型，注意shift right的操作。
  2. 动态学习率
from mindspore import ops
from mindnlp.transformers import GPT2LMHeadModel 
from mindspore.nn.learning_rate_schedule import LearningRateSchedulefrom mindspore import nn
from mindnlp.transformers import GPT2Config, GPT2LMHeadModel
from mindnlp._legacy.engine import Trainer
from mindnlp._legacy.engine.callbacks import CheckpointCallbackclass GPT2ForSummarization(GPT2LMHeadModel):def construct(self,input_ids = None,attention_mask = None,labels = None,):outputs = super().construct(input_ids=input_ids, attention_mask=attention_mask)shift_logits = outputs.logits[..., :-1, :]shift_labels = labels[..., 1:]# Flatten the tokensloss = ops.cross_entropy(shift_logits.view(-1, shift_logits.shape[-1]), shift_labels.view(-1), ignore_index=tokenizer.pad_token_id)return lossclass LinearWithWarmUp(LearningRateSchedule):"""Warmup-decay learning rate."""def __init__(self, learning_rate, num_warmup_steps, num_training_steps):super().__init__()self.learning_rate = learning_rateself.num_warmup_steps = num_warmup_stepsself.num_training_steps = num_training_stepsdef construct(self, global_step):if global_step < self.num_warmup_steps:return global_step / float(max(1, self.num_warmup_steps)) * self.learning_ratereturn ops.maximum(0.0, (self.num_training_steps - global_step) / (max(1, self.num_training_steps - self.num_warmup_steps))) * self.learning_rate## 训练参数设置
num_epochs = 1
warmup_steps = 2000
learning_rate = 1.5e-4num_training_steps = num_epochs * train_dataset.get_dataset_size()config = GPT2Config(vocab_size=len(tokenizer))
model = GPT2ForSummarization(config)lr_scheduler = LinearWithWarmUp(learning_rate=learning_rate, num_warmup_steps=warmup_steps, num_training_steps=num_training_steps)
optimizer = nn.AdamWeightDecay(model.trainable_params(), learning_rate=lr_scheduler)# 记录模型参数数量
print('number of model parameters: {}'.format(model.num_parameters()))

gpt2模型结构输出

1. 1级主类：GPT2ForSummarization

2. 2级类：GPT2Model 层，是transformer 结构，是模型的核心部分。

3. 2级类：lm_head 结构的 Dense 全连接层 , dim[in, out]=[768,  21128]。

4. GPT2Model 结构下的3级类组件分三层：

        >> wte 嵌入层：dim[in, out]=[21128, 768] ，即使用了 21128 个词汇，每个词汇映射到一个768 维的向量。

        >> wpe 嵌入层：dim[in, out]=[1024, 768] 

        >> drop 层。

        >> layers h 隐网络结构层：Transformer模型的主体，包含 12 个 GPT2Block。  

        >> ln_f LayerNorm 最后的层归一化。        

5. GPT2Block 的结构：

        》》ln_1 LayerNorm层，层归一化，用于在注意力机制之前对输入进行归一化。

        》》attn GPT2Attention层，自注意力机制，用于计算输入序列中不同位置的注意力权重。共包括3层：Conv1D、Conv1D、CustomDropout、CustomDropout。

        》》ln_2 LayerNorm层，用于自注意力之后的归一化。

        》》mlp  GPT2MLP层，多层感知机，用于对自注意力层的输出进行进一步的非线性变换。这里使用的操作包括：Conv1D、Conv1D、GELU、CustomDropout。
 

$ print(model)GPT2ForSummarization<(transformer): GPT2Model<(wte): Embedding<vocab_size=21128, embedding_size=768, use_one_hot=False, weight=Parameter (Tensor(shape=[21128, 768], dtype=Float32, value=[...], name=transformer.wte.weight), requires_grad=True), dtype=Float32, padding_idx=None>(wpe): Embedding<vocab_size=1024, embedding_size=768, use_one_hot=False, weight=Parameter (Tensor(shape=[1024, 768], dtype=Float32, value=[...], name=transformer.wpe.weight), requires_grad=True), dtype=Float32, padding_idx=None>(drop): CustomDropout<>(h): CellList<(0): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.0.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(1): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.1.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(2): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.2.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(3): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.3.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(4): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.4.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(5): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.5.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(6): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.6.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(7): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.7.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(8): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.8.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(9): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.9.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(10): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.10.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>(11): GPT2Block<(ln_1): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_1.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_1.bias), requires_grad=True)>(attn): GPT2Attention<(c_attn): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(attn_dropout): CustomDropout<>(resid_dropout): CustomDropout<>>(ln_2): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_2.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.h.11.ln_2.bias), requires_grad=True)>(mlp): GPT2MLP<(c_fc): Conv1D<(matmul): Matmul<>>(c_proj): Conv1D<(matmul): Matmul<>>(act): GELU<>(dropout): CustomDropout<>>>>(ln_f): LayerNorm<normalized_shape=[768], begin_norm_axis=-1, begin_params_axis=-1, weight=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.ln_f.weight), requires_grad=True), bias=Parameter (Tensor(shape=[768], dtype=Float32, value=[...], name=transformer.ln_f.bias), requires_grad=True)>>(lm_head): Dense<input_channels=768, output_channels=21128>>

训练流程

from mindspore import nn
from mindnlp.transformers import GPT2Config, GPT2LMHeadModel
from mindnlp._legacy.engine import Trainer
from mindnlp._legacy.engine.callbacks import CheckpointCallback# 记录模型参数数量
print('number of model parameters: {}'.format(model.num_parameters()))ckpoint_cb = CheckpointCallback(save_path='checkpoint', ckpt_name='gpt2_summarization',epochs=1, keep_checkpoint_max=2)trainer = Trainer(network=model, train_dataset=train_dataset,epochs=1, optimizer=optimizer, callbacks=ckpoint_cb)
trainer.set_amp(level='O1')  # 开启混合精度trainer.run(tgt_columns="labels")

部分输出

注：建议使用较高规格的算力，训练时间较长

部分输出2（减少训练数据）

此次活动的 notebook 只可以连续运行8小时，此次目的也不是性能优化，故此，我将训练数据减少到了1/10，此时的部分输出如下。

推理流程

## 向量数据转为中文数据
def process_test_dataset(dataset, tokenizer, batch_size=1, max_seq_len=1024, max_summary_len=100):def read_map(text):data = json.loads(text.tobytes())return np.array(data['article']), np.array(data['summarization'])def pad(article):tokenized = tokenizer(text=article, truncation=True, max_length=max_seq_len-max_summary_len)return tokenized['input_ids']dataset = dataset.map(read_map, 'text', ['article', 'summary'])dataset = dataset.map(pad, 'article', ['input_ids'])dataset = dataset.batch(batch_size)return datasettest_dataset = process_test_dataset(test_dataset, tokenizer, batch_size=1)
print(next(test_dataset.create_tuple_iterator(output_numpy=True)))model = GPT2LMHeadModel.from_pretrained('./checkpoint/gpt2_summarization_epoch_0.ckpt', config=config)model.set_train(False)
model.config.eos_token_id = model.config.sep_token_id
i = 0
for (input_ids, raw_summary) in test_dataset.create_tuple_iterator():output_ids = model.generate(input_ids, max_new_tokens=50, num_beams=5, no_repeat_ngram_size=2)output_text = tokenizer.decode(output_ids[0].tolist())print(output_text)i += 1if i == 1:break

减少训练数据后的模型推理结果展示。

http://www.yayakq.cn/news/662097/

相关文章：

  • 做外贸的如何上国外网站wordpress二维码分享
  • 内网门户网站wordpress屏蔽评论
  • 什么是企业型网站wordpress的.htaccess
  • 那些网站权重高珠海移动互联网开发
  • 怎么做网站切图前端开发有哪些
  • 举重运动员 做网站海南人才网
  • 用源码做自己的网站无线网站应建设在什么地方
  • 网页设计工作岗位及薪资直通车优化推广
  • 网站接做网站单一对一视频软件开发
  • 鄂州网站建设推广报价400免费服务电话申请
  • 网站怎么做站长统计南通专业家纺网站建设
  • 东莞网站开发多少钱旅行社网站模板
  • 怎么在国税网站上做实名认证php企业中英文网站源码
  • 精利手表网站电子商务网站建设与管理实验报告
  • 企业网站建设优化企业网站建设可行分析
  • 建设银行河北分行官网招聘网站国内最好的在线网站建设
  • 上海源码网站建设公司wordpress搭建电商
  • 网站域名优化在线网页刷新
  • 社交网站是怎么做的做网站主题
  • 网站后台管理系统展望设计网站大全软件
  • 个人网站的订单天津基础设施建设
  • 去视频网站做编辑器erp企业管理软件
  • 网站建设推广关键词wordpress 跑马灯
  • 绿色企业网站建立多个wordpress
  • 想要弄一个网站怎么弄网站seo优化检测
  • 超好看WordPress网站制作和优化
  • 怎么做网站赚流量给了几张图让做网站
  • 做网站推广那家好destoon 网站搬迁
  • 凡科建设网站入门版好不注册公司费用流程图
  • 晋城客运东站网站开发ui网站设计模板