k8s集群的日志，带有组件的信息，多看日志。
kubectl命令汇总
kubectl命令汇总
kubectl命令帮助信息
[root@mcwk8s04 ~]# kubectl -h kubectl controls the Kubernetes cluster manager. Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/ Basic Commands (Beginner): create Create a resource from a file or from stdin. expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service run Run a particular image on the cluster set Set specific features on objects Basic Commands (Intermediate): explain Documentation of resources get Display one or many resources edit Edit a resource on the server delete Delete resources by filenames, stdin, resources and names, or by resources and label selector Deploy Commands: rollout Manage the rollout of a resource scale Set a new size for a Deployment, ReplicaSet, Replication Controller, or Job autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController Cluster Management Commands: certificate Modify certificate resources. cluster-info Display cluster info top Display Resource (CPU/Memory/Storage) usage. cordon Mark node as unschedulable uncordon Mark node as schedulable drain Drain node in preparation for maintenance taint Update the taints on one or more nodes Troubleshooting and Debugging Commands: describe Show details of a specific resource or group of resources logs Print the logs for a container in a pod attach Attach to a running container exec Execute a command in a container port-forward Forward one or more local ports to a pod proxy Run a proxy to the Kubernetes API server cp Copy files and directories to and from containers. auth Inspect authorization Advanced Commands: diff Diff live version against would-be applied version apply Apply a configuration to a resource by filename or stdin patch Update field(s) of a resource using strategic merge patch replace Replace a resource by filename or stdin wait Experimental: Wait for a specific condition on one or many resources. convert Convert config files between different API versions kustomize Build a kustomization target from a directory or a remote url. Settings Commands: label Update the labels on a resource annotate Update the annotations on a resource completion Output shell completion code for the specified shell (bash or zsh) Other Commands: api-resources Print the supported API resources on the server api-versions Print the supported API versions on the server, in the form of "group/version" config Modify kubeconfig files plugin Provides utilities for interacting with plugins. version Print the client and server version information Usage: kubectl [flags] [options] Use "kubectl <command> --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). [root@mcwk8s04 ~]#
kubectl run 命令帮助信息
[root@mcwk8s04 ~]# kubectl run --help
Create and run a particular image, possibly replicated.
Creates a deployment or job to manage the created container(s).
Examples:
# Start a single instance of nginx.
kubectl run nginx --image=nginx
# Start a single instance of hazelcast and let the container expose port 5701 .
kubectl run hazelcast --image=hazelcast --port=5701
# Start a single instance of hazelcast and set environment variables "DNS_DOMAIN=cluster" and "POD_NAMESPACE=default"
in the container.
kubectl run hazelcast --image=hazelcast --env="DNS_DOMAIN=cluster" --env="POD_NAMESPACE=default"
# Start a single instance of hazelcast and set labels "app=hazelcast" and "env=prod" in the container.
kubectl run hazelcast --image=hazelcast --labels="app=hazelcast,env=prod"
# Start a replicated instance of nginx.
kubectl run nginx --image=nginx --replicas=5
# Dry run. Print the corresponding API objects without creating them.
kubectl run nginx --image=nginx --dry-run
# Start a single instance of nginx, but overload the spec of the deployment with a partial set of values parsed from
JSON.
kubectl run nginx --image=nginx --overrides='{ "apiVersion": "v1", "spec": { ... } }'
# Start a pod of busybox and keep it in the foreground, don't restart it if it exits.
kubectl run -i -t busybox --image=busybox --restart=Never
# Start the nginx container using the default command, but use custom arguments (arg1 .. argN) for that command.
kubectl run nginx --image=nginx -- <arg1> <arg2> ... <argN>
# Start the nginx container using a different command and custom arguments.
kubectl run nginx --image=nginx --command -- <cmd> <arg1> ... <argN>
# Start the perl container to compute π to 2000 places and print it out.
kubectl run pi --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
# Start the cron job to compute π to 2000 places and print it out every 5 minutes.
kubectl run pi --schedule="0/5 * * * ?" --image=perl --restart=OnFailure -- perl -Mbignum=bpi -wle 'print bpi(2000)'
Options:
--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in
the template. Only applies to golang and jsonpath output formats.
--attach=false: If true, wait for the Pod to start running, and then attach to the Pod as if 'kubectl attach ...'
were called. Default false, unless '-i/--stdin' is set, in which case the default is true. With '--restart=Never' the
exit code of the container process is returned.
--cascade=true: If true, cascade the deletion of the resources managed by this resource (e.g. Pods created by a
ReplicationController). Default true.
--command=false: If true and extra arguments are present, use them as the 'command' field in the container, rather
than the 'args' field which is the default.
--dry-run=false: If true, only print the object that would be sent, without sending it.
--env=[]: Environment variables to set in the container
--expose=false: If true, a public, external service is created for the container(s) which are run
-f, --filename=[]: to use to replace the resource.
--force=false: Only used when grace-period=0. If true, immediately remove resources from API and bypass graceful
deletion. Note that immediate deletion of some resources may result in inconsistency or data loss and requires
confirmation.
--generator='': The name of the API generator to use, see
http://kubernetes.io/docs/user-guide/kubectl-conventions/#generators for a list.
--grace-period=-1: Period of time in seconds given to the resource to terminate gracefully. Ignored if negative.
Set to 1 for immediate shutdown. Can only be set to 0 when --force is true (force deletion).
--hostport=-1: The host port mapping for the container port. To demonstrate a single-machine container.
--image='': The image for the container to run.
--image-pull-policy='': The image pull policy for the container. If left empty, this value will not be specified
by the client and defaulted by the server
-k, --kustomize='': Process a kustomization directory. This flag can't be used together with -f or -R.
-l, --labels='': Comma separated labels to apply to the pod(s). Will override previous values.
--leave-stdin-open=false: If the pod is started in interactive mode or with stdin, leave stdin open after the
first attach completes. By default, stdin will be closed after the first attach completes.
--limits='': The resource requirement limits for this container. For example, 'cpu=200m,memory=512Mi'. Note that
server side components may assign limits depending on the server configuration, such as limit ranges.
-o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.
--overrides='': An inline JSON override for the generated object. If this is non-empty, it is used to override the
generated object. Requires that the object supply a valid apiVersion field.
--pod-running-timeout=1m0s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
pod is running
--port='': The port that this container exposes. If --expose is true, this is also the port used by the service
that is created.
--quiet=false: If true, suppress prompt messages.
--record=false: Record current kubectl command in the resource annotation. If set to false, do not record the
command. If set to true, record the command. If not set, default to updating the existing annotation value only if one
already exists.
-R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage
related manifests organized within the same directory.
-r, --replicas=1: Number of replicas to create for this container. Default is 1.
--requests='': The resource requirement requests for this container. For example, 'cpu=100m,memory=256Mi'. Note
that server side components may assign requests depending on the server configuration, such as limit ranges.
--restart='Always': The restart policy for this Pod. Legal values [Always, OnFailure, Never]. If set to 'Always'
a deployment is created, if set to 'OnFailure' a job is created, if set to 'Never', a regular pod is created. For the
latter two --replicas must be 1. Default 'Always', for CronJobs `Never`.
--rm=false: If true, delete resources created in this command for attached containers.
--save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the
annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
--schedule='': A schedule in the Cron format the job should be run with.
--service-generator='service/v2': The name of the generator to use for creating a service. Only used if --expose
is true
--service-overrides='': An inline JSON override for the generated service object. If this is non-empty, it is used
to override the generated object. Requires that the object supply a valid apiVersion field. Only used if --expose is
true.
--serviceaccount='': Service account to set in the pod spec
-i, --stdin=false: Keep stdin open on the container(s) in the pod, even if nothing is attached.
--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The
template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
--timeout=0s: The length of time to wait before giving up on a delete, zero means determine a timeout from the
size of the object
-t, --tty=false: Allocated a TTY for each container in the pod.
--wait=false: If true, wait for resources to be gone before returning. This waits for finalizers.
Usage:
kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool]
[--overrides=inline-json] [--command] -- [COMMAND] [args...] [options]
Use "kubectl options" for a list of global command-line options (applies to all commands).
[root@mcwk8s04 ~]#
kubectl expose帮助信息
[root@mcwk8s04 ~]# kubectl expose -h
Expose a resource as a new Kubernetes service.
Looks up a deployment, service, replica set, replication controller or pod by name and uses the selector for that
resource as the selector for a new service on the specified port. A deployment or replica set will be exposed as a
service only if its selector is convertible to a selector that service supports, i.e. when the selector contains only
the matchLabels component. Note that if no port is specified via --port and the exposed resource has multiple ports, all
will be re-used by the new service. Also if no labels are specified, the new service will re-use the labels from the
resource it exposes.
Possible resources include (case insensitive):
pod (po), service (svc), replicationcontroller (rc), deployment (deploy), replicaset (rs)
Examples:
# Create a service for a replicated nginx, which serves on port 80 and connects to the containers on port 8000.
kubectl expose rc nginx --port=80 --target-port=8000
# Create a service for a replication controller identified by type and name specified in "nginx-controller.yaml",
which serves on port 80 and connects to the containers on port 8000.
kubectl expose -f nginx-controller.yaml --port=80 --target-port=8000
# Create a service for a pod valid-pod, which serves on port 444 with the name "frontend"
kubectl expose pod valid-pod --port=444 --name=frontend
# Create a second service based on the above service, exposing the container port 8443 as port 443 with the name
"nginx-https"
kubectl expose service nginx --port=443 --target-port=8443 --name=nginx-https
# Create a service for a replicated streaming application on port 4100 balancing UDP traffic and named 'video-stream'.
kubectl expose rc streamer --port=4100 --protocol=UDP --name=video-stream
# Create a service for a replicated nginx using replica set, which serves on port 80 and connects to the containers on
port 8000.
kubectl expose rs nginx --port=80 --target-port=8000
# Create a service for an nginx deployment, which serves on port 80 and connects to the containers on port 8000.
kubectl expose deployment nginx --port=80 --target-port=8000
Options:
--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in
the template. Only applies to golang and jsonpath output formats.
--cluster-ip='': ClusterIP to be assigned to the service. Leave empty to auto-allocate, or set to 'None' to create
a headless service.
--dry-run=false: If true, only print the object that would be sent, without sending it.
--external-ip='': Additional external IP address (not managed by Kubernetes) to accept for the service. If this IP
is routed to a node, the service can be accessed by this IP in addition to its generated service IP.
-f, --filename=[]: Filename, directory, or URL to files identifying the resource to expose a service
--generator='service/v2': The name of the API generator to use. There are 2 generators: 'service/v1' and
'service/v2'. The only difference between them is that service port in v1 is named 'default', while it is left unnamed
in v2. Default is 'service/v2'.
-k, --kustomize='': Process the kustomization directory. This flag can't be used together with -f or -R.
-l, --labels='': Labels to apply to the service created by this call.
--load-balancer-ip='': IP to assign to the LoadBalancer. If empty, an ephemeral IP will be created and used
(cloud-provider specific).
--name='': The name for the newly created object.
-o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.
--overrides='': An inline JSON override for the generated object. If this is non-empty, it is used to override the
generated object. Requires that the object supply a valid apiVersion field.
--port='': The port that the service should serve on. Copied from the resource being exposed, if unspecified
--protocol='': The network protocol for the service to be created. Default is 'TCP'.
--record=false: Record current kubectl command in the resource annotation. If set to false, do not record the
command. If set to true, record the command. If not set, default to updating the existing annotation value only if one
already exists.
-R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage
related manifests organized within the same directory.
--save-config=false: If true, the configuration of current object will be saved in its annotation. Otherwise, the
annotation will be unchanged. This flag is useful when you want to perform kubectl apply on this object in the future.
--selector='': A label selector to use for this service. Only equality-based selector requirements are supported.
If empty (the default) infer the selector from the replication controller or replica set.)
--session-affinity='': If non-empty, set the session affinity for the service to this; legal values: 'None',
'ClientIP'
--target-port='': Name or number for the port on the container that the service should direct traffic to.
Optional.
--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The
template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
--type='': Type for this service: ClusterIP, NodePort, LoadBalancer, or ExternalName. Default is 'ClusterIP'.
Usage:
kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP|SCTP] [--target-port=number-or-name]
[--name=name] [--external-ip=external-ip-of-service] [--type=type] [options]
Use "kubectl options" for a list of global command-line options (applies to all commands).
[root@mcwk8s04 ~]#
kubectl logs帮助信息
[root@mcwk8s04 ~]# kubectl logs -h
Print the logs for a container in a pod or specified resource. If the pod has only one container, the container name is
optional.
Aliases:
logs, log
Examples:
# Return snapshot logs from pod nginx with only one container
kubectl logs nginx
# Return snapshot logs from pod nginx with multi containers
kubectl logs nginx --all-containers=true
# Return snapshot logs from all containers in pods defined by label app=nginx
kubectl logs -lapp=nginx --all-containers=true
# Return snapshot of previous terminated ruby container logs from pod web-1
kubectl logs -p -c ruby web-1
# Begin streaming the logs of the ruby container in pod web-1
kubectl logs -f -c ruby web-1
# Begin streaming the logs from all containers in pods defined by label app=nginx
kubectl logs -f -lapp=nginx --all-containers=true
# Display only the most recent 20 lines of output in pod nginx
kubectl logs --tail=20 nginx
# Show all logs from pod nginx written in the last hour
kubectl logs --since=1h nginx
# Return snapshot logs from first container of a job named hello
kubectl logs job/hello
# Return snapshot logs from container nginx-1 of a deployment named nginx
kubectl logs deployment/nginx -c nginx-1
Options:
--all-containers=false: Get all containers' logs in the pod(s).
-c, --container='': Print the logs of this container
-f, --follow=false: Specify if the logs should be streamed.
--ignore-errors=false: If watching / following pod logs, allow for any errors that occur to be non-fatal
--limit-bytes=0: Maximum bytes of logs to return. Defaults to no limit.
--max-log-requests=5: Specify maximum number of concurrent logs to follow when using by a selector. Defaults to 5.
--pod-running-timeout=20s: The length of time (like 5s, 2m, or 3h, higher than zero) to wait until at least one
pod is running
-p, --previous=false: If true, print the logs for the previous instance of the container in a pod if it exists.
-l, --selector='': Selector (label query) to filter on.
--since=0s: Only return logs newer than a relative duration like 5s, 2m, or 3h. Defaults to all logs. Only one of
since-time / since may be used.
--since-time='': Only return logs after a specific date (RFC3339). Defaults to all logs. Only one of since-time /
since may be used.
--tail=-1: Lines of recent log file to display. Defaults to -1 with no selector, showing all log lines otherwise
10, if a selector is provided.
--timestamps=false: Include timestamps on each line in the log output
Usage:
kubectl logs [-f] [-p] (POD | TYPE/NAME) [-c CONTAINER] [options]
Use "kubectl options" for a list of global command-line options (applies to all commands).
[root@mcwk8s04 ~]#
kubectl set 帮助信息
[root@mcwk8s03 ~]# kubectl set -h Configure application resources These commands help you make changes to existing application resources. Available Commands: env Update environment variables on a pod template image Update image of a pod template resources Update resource requests/limits on objects with pod templates selector Set the selector on a resource serviceaccount Update ServiceAccount of a resource subject Update User, Group or ServiceAccount in a RoleBinding/ClusterRoleBinding Usage: kubectl set SUBCOMMAND [options] Use "kubectl <command> --help" for more information about a given command. Use "kubectl options" for a list of global command-line options (applies to all commands). [root@mcwk8s03 ~]#
kubectl set image帮助信息
[root@mcwk8s03 ~]# kubectl set image -h
Update existing container image(s) of resources.
Possible resources include (case insensitive):
pod (po), replicationcontroller (rc), deployment (deploy), daemonset (ds), replicaset (rs)
Examples:
# Set a deployment's nginx container image to 'nginx:1.9.1', and its busybox container image to 'busybox'.
kubectl set image deployment/nginx busybox=busybox nginx=nginx:1.9.1
# Update all deployments' and rc's nginx container's image to 'nginx:1.9.1'
kubectl set image deployments,rc nginx=nginx:1.9.1 --all
# Update image of all containers of daemonset abc to 'nginx:1.9.1'
kubectl set image daemonset abc *=nginx:1.9.1
# Print result (in yaml format) of updating nginx container image from local file, without hitting the server
kubectl set image -f path/to/file.yaml nginx=nginx:1.9.1 --local -o yaml
Options:
--all=false: Select all resources, including uninitialized ones, in the namespace of the specified resource types
--allow-missing-template-keys=true: If true, ignore any errors in templates when a field or map key is missing in
the template. Only applies to golang and jsonpath output formats.
--dry-run=false: If true, only print the object that would be sent, without sending it.
-f, --filename=[]: Filename, directory, or URL to files identifying the resource to get from a server.
-k, --kustomize='': Process the kustomization directory. This flag can't be used together with -f or -R.
--local=false: If true, set image will NOT contact api-server but run locally.
-o, --output='': Output format. One of:
json|yaml|name|go-template|go-template-file|template|templatefile|jsonpath|jsonpath-file.
--record=false: Record current kubectl command in the resource annotation. If set to false, do not record the
command. If set to true, record the command. If not set, default to updating the existing annotation value only if one
already exists.
-R, --recursive=false: Process the directory used in -f, --filename recursively. Useful when you want to manage
related manifests organized within the same directory.
-l, --selector='': Selector (label query) to filter on, not including uninitialized ones, supports '=', '==', and
'!='.(e.g. -l key1=value1,key2=value2)
--template='': Template string or path to template file to use when -o=go-template, -o=go-template-file. The
template format is golang templates [http://golang.org/pkg/text/template/#pkg-overview].
Usage:
kubectl set image (-f FILENAME | TYPE NAME) CONTAINER_NAME_1=CONTAINER_IMAGE_1 ... CONTAINER_NAME_N=CONTAINER_IMAGE_N
[options]
Use "kubectl options" for a list of global command-line options (applies to all commands).
[root@mcwk8s03 ~]#
kubectl部署应用的周期
通过kubectl部署一个应用，他的生命周期如下，会经历如下过程。创建资源，发布应用，应用更新，应用回滚，以及应用删除。我们的应用一般都是直接构建好的镜像，然后直接就能部署。应用可以实现多个副本。
1、创建 kubectl run nginx --replicas=3 --image=nginx:1.14 --port=80 kubectl get deploy,pods 2、发布 kubectl expose deployment nginx --port=80 --type=NodePort --target-port=80 --name=nginx-service kubectl get service 3、更新 kubectl set image deployment/nginx nginx=nginx:1.15 4、回滚 kubectl rollout history deployment/nginx kubectl rollout undo deployment/nginx 5、删除 kubectl delete deploy/nginx kubectl delete svc/nginx-service
kubectl部署应用过程详解
kubectl run 运行容器
kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] [--command] -- [COMMAND] [args...] [options]
NAM：容器的名称
--image：指定使用的镜像
--env：指定容器内的环境变量，
--port：指定容器中需要映射的端口，也就是需要暴露的服务的端口
--replicas：指定副本数
--dry-run：指定是创建还是测试用
--overrides：可以重写配置
--command ：可以替换容器启动时的命令。
[root@mcwk8s04 ~]# [root@mcwk8s04 ~]# kubectl run mcw-nginx-deployment --image=nginx --port=80 --replicas=3 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/mcw-nginx-deployment created [root@mcwk8s04 ~]# [root@mcwk8s04 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mcw-nginx-deployment-86466dbd78-fkrs7 1/1 Running 0 27m 172.17.13.2 mcwk8s05 <none> <none> mcw-nginx-deployment-86466dbd78-rnvsc 1/1 Running 0 27m 172.17.13.4 mcwk8s05 <none> <none> mcw-nginx-deployment-86466dbd78-znjbm 1/1 Running 0 27m 172.17.21.3 mcwk8s06 <none> <none> [root@mcwk8s04 ~]#
kubectl get
指定查看多个资源
[root@mcwk8s04 ~]# kubectl get deploy,pods NAME READY UP-TO-DATE AVAILABLE AGE deployment.extensions/mcw-nginx-deployment 3/3 3 3 29m NAME READY STATUS RESTARTS AGE pod/mcw-nginx-deployment-86466dbd78-fkrs7 1/1 Running 0 29m pod/mcw-nginx-deployment-86466dbd78-rnvsc 1/1 Running 0 29m pod/mcw-nginx-deployment-86466dbd78-znjbm 1/1 Running 0 29m [root@mcwk8s04 ~]#
查看副本
[root@mcwk8s04 ~]# kubectl get replicasets NAME DESIRED CURRENT READY AGE mcw-nginx-deployment-86466dbd78 3 3 3 31m [root@mcwk8s04 ~]# kubectl get rs NAME DESIRED CURRENT READY AGE mcw-nginx-deployment-86466dbd78 3 3 3 31m [root@mcwk8s04 ~]#
kubectl expose 让pod提供对外服务
kubectl expose (-f FILENAME | TYPE NAME) [--port=port] [--protocol=TCP|UDP|SCTP] [--target-port=number-or-name] [--name=name] [--external-ip=external-ip-of-service] [--type=type] [options] -f FILENAME | TYPE NAME ：可以是yaml文件，指定资源类型。
--port：指定暴露的端口，集群内访问service的端口--protocol：指定service转发的协议类型--target-port：容器内服务端口--name ：指定service的名字--type：指定service类型 ClusterIP, NodePort, LoadBalancer, or ExternalName
[root@mcwk8s04 ~]# kubectl expose deployment --port=80 --target-port=80 --name=mcw-nginx-service --type=NodePort error: resource(s) were provided, but no name, label selector, or --all flag specified See 'kubectl expose -h' for help and examples [root@mcwk8s04 ~]# kubectl get deployment NAME READY UP-TO-DATE AVAILABLE AGE mcw-nginx-deployment 3/3 3 3 54m [root@mcwk8s04 ~]# kubectl expose deployment mcw-nginx-deployment --port=80 --target-port=80 --name=mcw-nginx-service --type=NodePort service/mcw-nginx-service exposed [root@mcwk8s04 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 5d19h mcw-nginx-service NodePort 10.2.0.252 <none> 80:39719/TCP 11s [root@mcwk8s04 ~]#
kubectl api-resources资源简写
[root@mcwk8s04 ~]# kubectl api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node persistentvolumeclaims pvc true PersistentVolumeClaim persistentvolumes pv false PersistentVolume pods po true Pod podtemplates true PodTemplate replicationcontrollers rc true ReplicationController resourcequotas quota true ResourceQuota secrets true Secret serviceaccounts sa true ServiceAccount services svc true Service mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition apiservices apiregistration.k8s.io false APIService controllerrevisions apps true ControllerRevision daemonsets ds apps true DaemonSet deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet tokenreviews authentication.k8s.io false TokenReview localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview subjectaccessreviews authorization.k8s.io false SubjectAccessReview horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler cronjobs cj batch true CronJob jobs batch true Job certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest leases coordination.k8s.io true Lease events ev events.k8s.io true Event daemonsets ds extensions true DaemonSet deployments deploy extensions true Deployment ingresses ing extensions true Ingress networkpolicies netpol extensions true NetworkPolicy podsecuritypolicies psp extensions false PodSecurityPolicy replicasets rs extensions true ReplicaSet ingresses ing networking.k8s.io true Ingress networkpolicies netpol networking.k8s.io true NetworkPolicy runtimeclasses node.k8s.io false RuntimeClass poddisruptionbudgets pdb policy true PodDisruptionBudget podsecuritypolicies psp policy false PodSecurityPolicy clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding clusterroles rbac.authorization.k8s.io false ClusterRole rolebindings rbac.authorization.k8s.io true RoleBinding roles rbac.authorization.k8s.io true Role priorityclasses pc scheduling.k8s.io false PriorityClass csidrivers storage.k8s.io false CSIDriver csinodes storage.k8s.io false CSINode storageclasses sc storage.k8s.io false StorageClass volumeattachments storage.k8s.io false VolumeAttachment [root@mcwk8s04 ~]#
service的三种端口
portservice暴露在cluster ip上的端口，:port 是提供给集群内部客户访问service的入口。
nodePortnodePort是k8s提供给集群外部客户访问service入口的一种方式，:nodePort 是提供给集群外部客户访问service的入口。
targetPorttargetPort是pod上的端口，从port和nodePort上到来的数据最终经过kube-proxy流入到后端pod的targetPort上进入容器。
port、nodePort总结总的来说，port和nodePort都是service的端口，前者暴露给集群内客户访问服务，后者暴露给集群外客户访问服务。从这两个端口到来的数据都需要经过反向代理kube-proxy流入后端pod的targetPod，从而到达pod上的容器内。
查看service转发情况ipvadm
service NodePort 39719 和endpoint [root@mcwk8s03 ~]# kubectl get svc mcw-nginx-service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE mcw-nginx-service NodePort 10.2.0.252 <none> 80:39719/TCP 9m21s [root@mcwk8s03 ~]# kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES mcw-nginx-deployment-86466dbd78-fkrs7 1/1 Running 0 64m 172.17.13.2 mcwk8s05 <none> <none> mcw-nginx-deployment-86466dbd78-rnvsc 1/1 Running 0 64m 172.17.13.4 mcwk8s05 <none> <none> mcw-nginx-deployment-86466dbd78-znjbm 1/1 Running 0 64m 172.17.21.3 mcwk8s06 <none> <none> [root@mcwk8s03 ~]# [root@mcwk8s03 ~]# kubectl get endpoints mcw-nginx-service NAME ENDPOINTS AGE mcw-nginx-service 172.17.13.2:80,172.17.13.4:80,172.17.21.3:80 12m [root@mcwk8s03 ~]# node上查看nodeport [root@mcwk8s05 ~]# netstat -lntup|grep 39719 tcp6 0 0 :::39719 :::* LISTEN 130848/kube-proxy [root@mcwk8s05 ~]# node上使用ipvadm查看service的转发情况(负载均衡情况) [root@mcwk8s05 ~]# yum install ipvsadm -y [root@mcwk8s05 ~]# ipvsadm -L -n IP Virtual Server version 1.2.1 (size=4096) Prot LocalAddress:Port Scheduler Flags -> RemoteAddress:Port Forward Weight ActiveConn InActConn TCP 10.0.0.35:39719 rr #node ip : node port 就转发到下面三个pod的80服务上 -> 172.17.13.2:80 Masq 1 0 0 -> 172.17.13.4:80 Masq 1 0 0 -> 172.17.21.3:80 Masq 1 0 0 ...... TCP 172.17.13.0:39719 rr -> 172.17.13.2:80 Masq 1 0 0 -> 172.17.13.4:80 Masq 1 0 0 -> 172.17.21.3:80 Masq 1 0 0 ...... TCP 172.17.13.1:39719 rr -> 172.17.13.2:80 Masq 1 0 0 -> 172.17.13.4:80 Masq 1 0 0 -> 172.17.21.3:80 Masq 1 0 0 .......
TCP 10.2.0.252:80 rr #service的cluster ip : port 提供集群内部访问service用的，这个端口和宿主机端口没关系，不是宿主机端口 -> 172.17.13.2:80 Masq 1 0 0 -> 172.17.13.4:80 Masq 1 0 0 -> 172.17.21.3:80 Masq 1 0 0
[root@mcwk8s05 ~]#
浏览器上访问时
谷歌浏览器成功访问，被调度到其中一个pod上了
kubectl set 更新应用
使用k8s，使用的是镜像。我们需要更新应用，就是换新的镜像。kubectl set可以替换镜像。它还有一些子命令env Update environment variables on a pod templateimage Update image of a pod templateresources 可以对资源，比如cpu等进行限制，Update resource requests/limits on objects with pod templatesselector 修改标签，Set the selector on a resourceserviceaccount Update ServiceAccount of a resourcesubject 角色绑定 。Update User, Group or ServiceAccount in a RoleBinding/ClusterRoleBinding
kubectl -h
kubectl set -h
kubectl set image -h
我们可以层层去看帮助信息，找到我们需要的命令。
比如我们就用下面的改改，
当我们请求网站的时候，响应头里面带了服务的版本信息
我们将nginx改为1.14版本的镜像
kubectl set image deployment/nginx busybox=busybox nginx=nginx:1.9.1
根据帮助信息里的修改，deployment名字要对上。后面容器名字=新的镜像要对上容器名，把pod后面的随机字符串去掉就行。貌似也就是deployment名称呀。-w会实时显示pod的情况
不知道为啥，master上执行命令，拉取镜像很慢，拉取不下来，但是我直接取node上docker pull很快的。可能是node上直接用的配置文件里面的三个镜像地址，而kubectl去拉去镜像，用的是k8s集群里哪里设置的一个镜像地址吧？我记得之前设置了一个阿里云镜像地址的。所以我直接就在node上执行命令拉取镜像了。让pod不用显示拉取镜像这一个步骤了。
-w可能理解有误，我这直接去node上执行命令pull镜像之后，貌似已经运行了，但是-w这里还是那里卡着不动。
我们再次去浏览器上的响应头里看，发现已经成了我们修改的1.14版本了
我们层层查看，就用这个命令来查看版本
[root@mcwk8s04 ~]# kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE mcw-nginx-deployment 3/3 3 3 3h54m [root@mcwk8s04 ~]# kubectl rollout history deployment/mcw-nginx-deployment deployment.extensions/mcw-nginx-deployment REVISION CHANGE-CAUSE 1 <none> 2 <none> [root@mcwk8s04 ~]#
回滚到上一个版本以及查看回滚的状态
[root@mcwk8s04 ~]# kubectl rollout undo deployment/mcw-nginx-deployment deployment.extensions/mcw-nginx-deployment rolled back [root@mcwk8s04 ~]# kubectl rollout status deployment/mcw-nginx-deployment Waiting for deployment "mcw-nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 1 out of 3 new replicas have been updated... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 2 out of 3 new replicas have been updated... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 2 old replicas are pending termination... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "mcw-nginx-deployment" rollout to finish: 1 old replicas are pending termination... deployment "mcw-nginx-deployment" successfully rolled out [root@mcwk8s04 ~]# kubectl rollout status deployment/mcw-nginx-deployment deployment "mcw-nginx-deployment" successfully rolled out [root@mcwk8s04 ~]#
再刷新一下，页面上已经显示之前的那个版本了
删除资源
[root@mcwk8s04 ~]# kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE mcw-nginx-deployment 3/3 3 3 4h1m [root@mcwk8s04 ~]# kubectl delete deploy mcw-nginx-deployment deployment.extensions "mcw-nginx-deployment" deleted [root@mcwk8s04 ~]# kubectl get pod NAME READY STATUS RESTARTS AGE mcw-nginx-deployment-86466dbd78-8r5wr 0/1 Terminating 0 4m23s [root@mcwk8s04 ~]# kubectl get pod No resources found. [root@mcwk8s04 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 5d22h mcw-nginx-service NodePort 10.2.0.252 <none> 80:39719/TCP 3h6m [root@mcwk8s04 ~]# kubectl delete svc mcw-nginx-service service "mcw-nginx-service" deleted [root@mcwk8s04 ~]# kubectl get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 5d22h [root@mcwk8s04 ~]#
查看所有的接口的版本
apps/v1 v1是稳定版 v1beta1,2是测试版 。这是http资源。我们生产一般不用测试版本api接口
[root@mcwk8s04 ~]# kubectl api-versions admissionregistration.k8s.io/v1beta1 apiextensions.k8s.io/v1beta1 apiregistration.k8s.io/v1 apiregistration.k8s.io/v1beta1 apps/v1 apps/v1beta1 apps/v1beta2 authentication.k8s.io/v1 authentication.k8s.io/v1beta1 authorization.k8s.io/v1 authorization.k8s.io/v1beta1 autoscaling/v1 autoscaling/v2beta1 autoscaling/v2beta2 batch/v1 batch/v1beta1 certificates.k8s.io/v1beta1 coordination.k8s.io/v1 coordination.k8s.io/v1beta1 events.k8s.io/v1beta1 extensions/v1beta1 networking.k8s.io/v1 networking.k8s.io/v1beta1 node.k8s.io/v1beta1 policy/v1beta1 rbac.authorization.k8s.io/v1 rbac.authorization.k8s.io/v1beta1 scheduling.k8s.io/v1 scheduling.k8s.io/v1beta1 storage.k8s.io/v1 storage.k8s.io/v1beta1 v1 [root@mcwk8s04 ~]#
通过yaml配置文件部署服务
我们在官网里一层层找，下面是各种资源模板
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
这篇文档的这个部位，就是一个yaml的案例
将官网的复制过来，然后执行部署
[root@mcwk8s03 ~]# mkdir demo
[root@mcwk8s03 ~]# cd demo/
[root@mcwk8s03 demo]# vim mcw-nginx-deployment.yaml
[root@mcwk8s03 demo]# cat mcw-nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]# kubectl create -f mcw-nginx-deployment.yaml
deployment.apps/nginx-deployment created
[root@mcwk8s03 demo]# kubectl get pod
NAME READY STATUS RESTARTS AGE
nginx-deployment-7fd6966748-mxg8p 1/1 Running 0 5s
nginx-deployment-7fd6966748-pgfgq 0/1 ContainerCreating 0 5s
nginx-deployment-7fd6966748-wsfj4 1/1 Running 0 5s
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]#
官网里面看其它资源的案例
部署service
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]# vim mcw-nginx-service.yaml
[root@mcwk8s03 demo]# cat mcw-nginx-service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-service
labels:
app: nginx
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
selector:
app: nginx
[root@mcwk8s03 demo]# kubectl create -f mcw-nginx-service.yaml
service/nginx-service created
[root@mcwk8s03 demo]# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.2.0.1 <none> 443/TCP 5d22h
nginx-service NodePort 10.2.0.247 <none> 80:33851/TCP 4s
[root@mcwk8s03 demo]#
访问上面刚刚部署的服务，版本也是对上的
YAML配置文件管理资源 实用技巧
• 用run命令生成
kubectl run --image=nginx my-deploy -o yaml --dry-run > my-deploy.yaml
• 用get命令导出
kubectl get my-deploy/nginx -o=yaml --export > my-deploy.yaml
• Pod容器的字段拼写忘记了
kubectl explain pods.spec.containers
用run命令生成 --dry-run生成配置文件
[root@mcwk8s03 demo]# kubectl run mcw-nginx-deployment --image=nginx --port=80 --replicas=3 --dry-run #执行部署的命令后面接--dry-run，不会进行部署，可以检查是否有语法错误 kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead. deployment.apps/mcw-nginx-deployment created (dry run) [root@mcwk8s03 demo]# kubectl get pod NAME READY STATUS RESTARTS AGE nginx-deployment-7fd6966748-mxg8p 1/1 Running 0 36m nginx-deployment-7fd6966748-pgfgq 1/1 Running 0 36m nginx-deployment-7fd6966748-wsfj4 1/1 Running 0 36m [root@mcwk8s03 demo]# kubectl get deploy NAME READY UP-TO-DATE AVAILABLE AGE nginx-deployment 3/3 3 3 37m [root@mcwk8s03 demo]#
[root@mcwk8s03 demo]# kubectl run mcw-nginx-deployment --image=nginx --port=80 --repli=3 --dry-run #如这条命令，副本数参数写错了就报错了Error: unknown flag: --repli
Examples: # Start a single instance of nginx. kubectl run nginx --image=nginx
--dry-run 不执行 -o指定输出格式，然后追加到文件中。这样可以生成yaml配置文件，我们就不需要去网上找模板了。将导出的模板多余的部分删除掉就行了。我们也可以获取到json格式的数据
[root@mcwk8s03 demo]# kubectl run mcw-nginx-deployment --image=nginx --port=80 --replicas=3 --dry-run -o yaml >mcwTest.yaml
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
[root@mcwk8s03 demo]# cat mcwTest.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
creationTimestamp: null
labels:
run: mcw-nginx-deployment
name: mcw-nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
run: mcw-nginx-deployment
strategy: {}
template:
metadata:
creationTimestamp: null
labels:
run: mcw-nginx-deployment
spec:
containers:
- image: nginx
name: mcw-nginx-deployment
ports:
- containerPort: 80
resources: {}
status: {}
[root@mcwk8s03 demo]# kubectl run mcw-nginx-deployment --image=nginx --port=80 --replicas=3 --dry-run -o json >mcwTest.json
kubectl run --generator=deployment/apps.v1 is DEPRECATED and will be removed in a future version. Use kubectl run --generator=run-pod/v1 or kubectl create instead.
[root@mcwk8s03 demo]# cat mcwTest.json
{
"kind": "Deployment",
"apiVersion": "apps/v1",
"metadata": {
"name": "mcw-nginx-deployment",
"creationTimestamp": null,
"labels": {
"run": "mcw-nginx-deployment"
}
},
"spec": {
"replicas": 3,
"selector": {
"matchLabels": {
"run": "mcw-nginx-deployment"
}
},
"template": {
"metadata": {
"creationTimestamp": null,
"labels": {
"run": "mcw-nginx-deployment"
}
},
"spec": {
"containers": [
{
"name": "mcw-nginx-deployment",
"image": "nginx",
"ports": [
{
"containerPort": 80
}
],
"resources": {}
}
]
}
},
"strategy": {}
},
"status": {}
}
[root@mcwk8s03 demo]#
如下，大部分资源都可以用这种方式生成
[root@mcwk8s03 demo]# kubectl api-resources NAME SHORTNAMES APIGROUP NAMESPACED KIND bindings true Binding componentstatuses cs false ComponentStatus configmaps cm true ConfigMap endpoints ep true Endpoints events ev true Event limitranges limits true LimitRange namespaces ns false Namespace nodes no false Node persistentvolumeclaims pvc true PersistentVolumeClaim persistentvolumes pv false PersistentVolume pods po true Pod podtemplates true PodTemplate replicationcontrollers rc true ReplicationController resourcequotas quota true ResourceQuota secrets true Secret serviceaccounts sa true ServiceAccount services svc true Service mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition apiservices apiregistration.k8s.io false APIService controllerrevisions apps true ControllerRevision daemonsets ds apps true DaemonSet deployments deploy apps true Deployment replicasets rs apps true ReplicaSet statefulsets sts apps true StatefulSet tokenreviews authentication.k8s.io false TokenReview localsubjectaccessreviews authorization.k8s.io true LocalSubjectAccessReview selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview subjectaccessreviews authorization.k8s.io false SubjectAccessReview horizontalpodautoscalers hpa autoscaling true HorizontalPodAutoscaler cronjobs cj batch true CronJob jobs batch true Job certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest leases coordination.k8s.io true Lease events ev events.k8s.io true Event daemonsets ds extensions true DaemonSet deployments deploy extensions true Deployment ingresses ing extensions true Ingress networkpolicies netpol extensions true NetworkPolicy podsecuritypolicies psp extensions false PodSecurityPolicy replicasets rs extensions true ReplicaSet ingresses ing networking.k8s.io true Ingress networkpolicies netpol networking.k8s.io true NetworkPolicy runtimeclasses node.k8s.io false RuntimeClass poddisruptionbudgets pdb policy true PodDisruptionBudget podsecuritypolicies psp policy false PodSecurityPolicy clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding clusterroles rbac.authorization.k8s.io false ClusterRole rolebindings rbac.authorization.k8s.io true RoleBinding roles rbac.authorization.k8s.io true Role priorityclasses pc scheduling.k8s.io false PriorityClass csidrivers storage.k8s.io false CSIDriver csinodes storage.k8s.io false CSINode storageclasses sc storage.k8s.io false StorageClass volumeattachments storage.k8s.io false VolumeAttachment [root@mcwk8s03 demo]#
用get命令导出已部署资源的配置，然后我们也可以据此修改
把不熟悉的，没有的删除掉
[root@mcwk8s03 demo]# kubectl get deploy
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 51m
[root@mcwk8s03 demo]# kubectl get deploy/nginx-deployment --export -o yaml
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: null
generation: 1
labels:
app: nginx
name: nginx-deployment
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-deployment
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status: {}
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]#
[root@mcwk8s03 demo]# kubectl get deploy/nginx-deployment --export -o=yaml
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "1"
creationTimestamp: null
generation: 1
labels:
app: nginx
name: nginx-deployment
selfLink: /apis/extensions/v1beta1/namespaces/default/deployments/nginx-deployment
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: nginx
spec:
containers:
- image: nginx:1.14.2
imagePullPolicy: IfNotPresent
name: nginx
ports:
- containerPort: 80
protocol: TCP
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
status: {}
[root@mcwk8s03 demo]# kubectl get deploy/nginx-deployment --export -o=yaml >myGet.yaml
Flag --export has been deprecated, This flag is deprecated and will be removed in future.
[root@mcwk8s03 demo]# head -3 myGet.yaml
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
[root@mcwk8s03 demo]#
Pod容器的字段拼写忘记了explain
比如我们忘了哪个字段是怎么拼写了可以像下面这样找到
[root@mcwk8s03 demo]# kubectl explain pods.spec.containers
KIND: Pod
VERSION: v1
RESOURCE: containers <[]Object>
DESCRIPTION:
List of containers belonging to the pod. Containers cannot currently be
added or removed. There must be at least one container in a Pod. Cannot be
updated.
A single application container that you want to run within a pod.
FIELDS:
args <[]string>
Arguments to the entrypoint. The docker image's CMD is
标签：
留言评论