介绍K8S使用CRD进行开发的操作步骤。
CRD中的generate-client-codes
在使用CustomResources时通常会使用以下几个code-generators:
- deepcopy-gen—creates a method func (t* T) DeepCopy() *T for each type T
- client-gen—creates typed clientsets for CustomResource APIGroups
- informer-gen—creates informers for CustomResources which offer an event based interface to react on changes of CustomResources on the server
- lister-gen—creates listers for CustomResources which offer a read-only caching layer for GET and LIST requests.
代码生成器codegen
所有Kubernetes代码生成器都是在k8s.io/gengo之上实现的。它们共享许多公共命令行flag。基本上,所有的生成器都会得到一个输入包的列表(–input-dis),它们逐个type遍历这些包,并输出生成的代码。
关于生成的代码:
或者转到与输入文件相同的目录中,比如inepCopy-gen(带有–输出文件基“zz_generated.deepcopy”来定义文件名)。
或者,它们生成一个或多个输出包(带有–output-package),比如client-, informer- and lister-gen (通常生成在pkg/client目录下)。
k8s.io/code-generator提供了一个shell脚本generator-group.sh,用于调用生成器及其对CustomResources用例的所有特殊的小需求。在自己的项目中要做的全部工作归结为一个一行程序脚本命令,具体执行代码通常位于hack/update-codegen.sh内。
1
2
3
4
5
| vendor/k8s.io/code-generator/generate-groups.sh all \
github.com/xxx_project/crd-code-generation/pkg/client \
github.com/xxx_project/crd-code-generation/pkg/apis \
example.com:v1
|
所有的API都在pkg/api下面,clientsets, informers, and listers都是在pkg/client内部创建的。也就是说,pkg/client是完全自动生成的,与包含CustomResourcegolang类型的Typees.go文件旁边的ZZ_generated.deepcopy.go文件一样,都是完全生成的。这两者都不应该手动修改,而是通过运行
1
2
3
4
5
| hack/update-codegen.sh
# 或者
hack/update-gencode.sh
|
通常,在update-codegen.sh文件旁边还有一个hack/version-codegen.sh或hack/verify-gencode.sh脚本,如果生成的任何文件都不是最新的,它将以非零返回代码终止。这在CI脚本中非常有用:如果开发人员意外地修改了文件,或者文件只是过时了,CI会注意到并提示。
虽然代码生成器的某些行为是通过上面描述的命令行标志(特别是要处理的包)来控制的,但是更多的属性是通过您的golang文件中的tag标记来控制的。
有2种类型tag
- Global tags above package in doc.go
- Local tags above a type that is processed
通常,标记的形状为//+标记名称或//+标记名称=值,也就是说,它们被写入注释中。根据标签的不同,注释的位置可能很重要。有许多标记必须位于类型(或全局标记的包行)上方的注释中,其他标记必须与类型(Pr包行)分隔,中间至少有一行空行。最好遵循一个例子,并复制基本的形状。
全局标记-GLOBAL TAGS
全局标记在pkg/apis///doc.go中定义,如下
pkg/apis/example.com/v1/doc.go
1
2
3
4
5
6
| // +k8s:deepcopy-gen=package,register
// Package v1 is the v1 version of the API.
// +groupName=example.com
package v1
|
默认情况下,它告诉DeepCopy-gen为该包中的每一种类型创建深度复制方法。如果您的类型不是必需的或不需要的,则可以使用本地标记//+k8s选择退出这样的类型:// +k8s:deepcopy-gen=false。如果不启用包范围的深度复制,则必须选择通过//+k8s对每种所需类型进行深度复制:// +k8s:deepcopy-gen=true。
最后,//+groupName=example.com定义了完全限定的API组名。如果搞错了,客户端会产生错误的代码。请注意,此标记必须位于包上方的注释块中。
本地标记可以直接写在API type上,也可以写在它上面的第二个注释块中。下面是一个示例 types.go。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
|
// +genclient
// +genclient:noStatus
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// Database describes a database.
type Database struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec DatabaseSpec `json:"spec"`
}
// DatabaseSpec is the spec for a Foo resource
type DatabaseSpec struct {
User string `json:"user"`
Password string `json:"password"`
Encoding string `json:"encoding,omitempty"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// DatabaseList is a list of Database resources
type DatabaseList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata"`
Items []Database `json:"items"`
}
|
注意,默认情况下,我们已为所有类型启用了deepcopy ,也可有选择退出。不过,这些类型都是API类型,需要deepcopy。因此,我们不必在本例 types.go中打开或关闭deepcopy,而只需在doc.go中的包范围内打开或关闭deepcopy。
runtime.Object and DeepCopyObject
有一个特殊的deepcopy tag说明
1
2
| // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
需要为type添加这个tag
原因如下:
k8s.op/apimachinery of the master branch—you have hit the compiler error that the CustomResource type does not implement runtime.Object because DeepCopyObject() runtime.Object is not defined on your type
解决方法:只需将以下本地标记置于顶级api类型之上
1
2
3
| // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
|
In our example above both Database and DatabaseList are top-level types because they are used as runtime.Objects. As a rule of thumb, top-level types are those which have metav1.TypeMeta embedded. Also, those are the types which clients are create for using client-gen.
Note, that the // +k8s:deepcopy-gen:interfaces tag can and should also be used in cases where you define API types that have fields of some interface type, for example, field SomeInterface. Then // +k8s:deepcopy-gen:interfaces=example.com/pkg/apis/example.SomeInterface will lead to the generation of a DeepCopySomeInterface() SomeInterface method. This allows it to deepcopy those fields in a type-correct way.
最后,有许多标记可以控制client-gen,我们在我们的示例中看到了其中的两个标记。
1
2
3
| // +genclient
// +genclient:noStatus
|
第一个标记告诉Client-gen为该类型创建一个client(这始终是选择加入)。注意,不必也不应该将其置于API对象的列表类型之上。
1
2
3
4
5
6
7
8
9
10
11
12
13
| // +genclient:nonNamespaced
// +genclient:noVerbs
// +genclient:onlyVerbs=create,delete
// +genclient:skipVerbs=get,list,create,update,patch,delete,deleteCollection,watch
// +genclient:method=Create,verb=create,result=k8s.io/apimachinery/pkg/apis/meta/v1.Status
|
程序使用client调用crd接口
使用client类型的main函数
A Main Function Using the Types Clients
main.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| import (
...
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/client-go/tools/clientcmd"
examplecomclientset "github.com/openshift-evangelist/crd-code-generation/pkg/client/clientset/versioned"
)
var (
kuberconfig = flag.String("kubeconfig", "", "Path to a kubeconfig. Only required if out-of-cluster.")
master = flag.String("master", "", "The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.")
)
func main() {
flag.Parse()
cfg, err := clientcmd.BuildConfigFromFlags(*master, *kuberconfig)
if err != nil {
glog.Fatalf("Error building kubeconfig: %v", err)
}
exampleClient, err := examplecomclientset.NewForConfig(cfg)
if err != nil {
glog.Fatalf("Error building example clientset: %v", err)
}
list, err := exampleClient.ExampleV1().Databases("default").List(metav1.ListOptions{})
if err != nil {
glog.Fatalf("Error listing all databases: %v", err)
}
for _, db := range list.Items {
fmt.Printf("database %s with user %q\n", db.Name, db.Spec.User)
}
}
|
tpye.go 实例
pkg/apis/example.com/v1/types.go
注意标记注释于type定义块之间的空行
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
| package v1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// NodeCacheStatus represents the status of NodeCche.
type NodeCacheStatus struct {
// The number of 'Unknonw' NodeCache in this Node.
Unknown int32 `json:"unknown,omitempty" protobuf:"bytes,1,opt,name=unknown"`
// The number of 'Pending' NodeCache in this queue.
Pending int32 `json:"pending,omitempty" protobuf:"bytes,2,opt,name=pending"`
// The number of 'Running' NodeCache in this queue.
Running int32 `json:"running,omitempty" protobuf:"bytes,3,opt,name=running"`
}
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type NodeCache struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
Spec NodeCacheSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// The status of NCDataSet.
// +optional
Status NodeCacheStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}
type NodeCacheSpec struct {
// dataset list
//Datasets []DataSetSummary `json:"datasets,omitempty" protobuf:"bytes,1,opt,name=datasets"`
Datasets string `json:"datasets,omitempty" protobuf:"bytes,1,opt,name=datasets"`
// Disk size unit :GB
FreeSize int64 `json:"freesize,omitempty" protobuf:"bytes,2,opt,name=freesize"`
// Disk size unit :GB
AllocatableSize int64 `json:"allocatablesize,omitempty" protobuf:"bytes,2,opt,name=allocatablesize"`
}
//// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
//
//type DataSetSummary struct {
// Digest string `json:"digest,omitempty" protobuf:"bytes,1,opt,name=digest"`
// Size int64 `json:"size,omitempty" protobuf:"bytes,2,opt,name=size"`
// InUse bool `json:"inuse,omitempty" protobuf:"bytes,3,opt,name=inuse"`
//}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
// NodeCacheList is a collection of NodeCache.
type NodeCacheList struct {
metav1.TypeMeta `json:",inline"`
// Standard list metadata
// More info: https://git.k8s.io/community/contributors/devel/api-conventions.md#metadata
// +optional
metav1.ListMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
// items is the list of NCDataSet
Items []NodeCache `json:"items" protobuf:"bytes,2,rep,name=items"`
}
|
CRD开发步骤
1. 编写CRD YAML
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
| apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: nodecaches.example.com
spec:
group: example.com
names:
kind: NodeCache
listKind: NodeCacheList
plural: nodecaches
scope: Cluster
version: v1
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
properties:
datasets:
type: string
freesize:
format: int64
type: integer
allocatablesize:
format: int64
type: integer
type: object
status:
properties:
unknown:
format: int32
type: integer
pending:
format: int32
type: integer
running:
format: int32
type: integer
type: object
type: object
|
2. 创建项目目录
按照pkg/apis/$GROUP/$VERSION/形式创建项目目录,需要在该目录下编写crd相关的几个go文件
- doc.go
- register.go
- types.go
1
2
3
4
5
6
7
8
9
10
11
| pkg/apis/example.com/v1/
cd $GOPATH/src/github.com/goprojects/demo/pkg/apis
.
├── apis
│ └── example.com
│ └── v1
│ ├── doc.go
│ ├── register.go
│ ├── types.go
|
3. 编写doc.go
位置:pkg/apis/$GROUP/$VERSION/doc.go
1
2
3
4
5
6
| // +k8s:deepcopy-gen=package,register
// Package v1beta1 is the v1beta1 version of the API.
// +groupName=example.com
package v1beta1
|
4. 编写types.go
位置:pkg/apis/$GROUP/$VERSION/types.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
| package v1beta1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// NodeStatStatus represents the status of NodeCche.
type NodeStatStatus struct {
// The number of 'Unknonw' NodeStat in this Node.
Unknown int32 `json:"unknown,omitempty" protobuf:"bytes,1,opt,name=unknown"`
// The number of 'Pending' NodeStat in this queue.
Pending int32 `json:"pending,omitempty" protobuf:"bytes,2,opt,name=pending"`
// The number of 'Running' NodeStat in this queue.
Running int32 `json:"running,omitempty" protobuf:"bytes,3,opt,name=running"`
}
// +genclient
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type NodeStat struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty" protobuf:"bytes,1,opt,name=metadata"`
Spec NodeStatSpec `json:"spec,omitempty" protobuf:"bytes,2,opt,name=spec"`
// The status of NodeStat.
// +optional
Status NodeStatStatus `json:"status,omitempty" protobuf:"bytes,3,opt,name=status"`
}
// ...
// more code define
// ...
|
5. 编写register.go
位置:pkg/apis/$GROUP/$VERSION/register.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
| package v1beta1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
)
const (
// GroupName is the group name used in this package.
GroupName = "example.com"
// GroupVersion is the version of scheduling group
GroupVersion = "v1beta1"
)
// SchemeGroupVersion is group version used to register these objects
var SchemeGroupVersion = schema.GroupVersion{Group: GroupName, Version: GroupVersion}
// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
return SchemeGroupVersion.WithResource(resource).GroupResource()
}
var (
// localSchemeBuilder and AddToScheme will stay in k8s.io/kubernetes.
SchemeBuilder runtime.SchemeBuilder
localSchemeBuilder = &SchemeBuilder
AddToScheme = localSchemeBuilder.AddToScheme
)
func init() {
// We only register manually written functions here. The registration of the
// generated functions takes place in the generated files. The separation
// makes the code compile even when the generated files are missing.
localSchemeBuilder.Register(addKnownTypes)
}
// Adds the list of known types to api.Scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
scheme.AddKnownTypes(SchemeGroupVersion,
&NodeStat{},
&NodeStatList{},
)
metav1.AddToGroupVersion(scheme, SchemeGroupVersion)
return nil
}
|
6. 执行
- code-generator自动生成代码
安装
1
2
3
4
| # set custom gopath
GOPATH=/home/test/go_projects/crddemo
go get k8s.io/code-generator
go get k8s.io/apimachinery
|
执行shell脚本生成指定代码内容
1
2
3
4
5
| cd $GOPATH/src/k8s.io/code-generator
./generate-groups.sh all \
"github.com/bingerambo/demo/pkg/client" \
"github.com/bingerambo/demo/pkg/apis" \
foo:v1
|
demo脚本 crd-code-gen
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
# cd /home/wangb/go_projects/crddemo/src/github.com/bingerambo/crd-code-generation/hack && ll
total 12
-rwxr-x---. 1 root root 565 Oct 5 2020 custom-boilerplate.go.txt
-rwxr-x---. 1 root root 543 Oct 5 2020 update-codegen.sh
-rwxr-x---. 1 root root 693 Feb 28 2018 verify-codegen.sh
# cd home/wangb/go_projects/crddemo/src/github.com/bingerambo/crd-code-generation/hack
cat update-codegen.sh
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
GOPATH="/home/wangb/go_projects/crddemo"
SCRIPT_ROOT=$(dirname ${BASH_SOURCE})/..
CODEGEN_PKG=${CODEGEN_PKG:-$(cd ${SCRIPT_ROOT}; ls -d -1 ./vendor/k8s.io/code-generator 2>/dev/null || echo ${GOPATH}/src/k8s.io/code-generator)}
vendor/k8s.io/code-generator/generate-groups.sh all \
github.com/bingerambo/crd-code-generation/pkg/nodecache_client github.com/bingerambo/crd-code-generation/pkg/apis \
example.com:v1 \
--go-header-file ${SCRIPT_ROOT}/hack/custom-boilerplate.go.txt
|
7. 实例
nodestat-crd.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
| ```yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: nodestats.example.com
spec:
group: example.com
names:
kind: NodeStat
listKind: NodeStatList
plural: nodestats
scope: Cluster
version: v1beta1
validation:
openAPIV3Schema:
properties:
apiVersion:
type: string
kind:
type: string
metadata:
type: object
spec:
type: object
properties:
nodename:
type: string
# 利用率
utility:
type: object
properties:
name:
type: string
sumarry:
type: string
average:
format: float
type: number
devs:
type: array
items:
type: object
properties:
id:
type: string
data:
format: float
type: number
# 总量
total:
type: object
properties:
name:
type: string
sumarry:
type: string
average:
format: float
type: number
devs:
type: array
items:
type: object
properties:
id:
type: string
data:
format: float
type: number
status:
properties:
unknown:
format: int32
type: integer
pending:
format: int32
type: integer
running:
format: int32
type: integer
type: object
type: object
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
|
#### 示例脚本nodestat-update-codegen.sh
github.com/kubernetes-sigs/kube-batch/hack
cat nodestat-update-codegen.sh
```shell
#!/bin/bash
# Copyright 2019 The Kube-batch Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
SCRIPT_ROOT=$(dirname ${BASH_SOURCE})/..
CODEGEN_PKG=${SCRIPT_ROOT}/vendor/k8s.io/code-generator
# generate the code with:
# --output-base because this script should also be able to run inside the vendor dir of
# k8s.io/kubernetes. The output-base is needed for the generators to output into the vendor dir
# instead of the $GOPATH directly. For normal projects this can be dropped.
cd ${SCRIPT_ROOT}
bash ${CODEGEN_PKG}/generate-groups.sh "deepcopy,client,informer,lister" \
github.com/kubernetes-sigs/kube-batch/pkg/nodestat_client github.com/kubernetes-sigs/kube-batch/pkg/apis \
"example.com:v1beta1" \
--go-header-file ${SCRIPT_ROOT}/hack/custom-boilerplate.go.txt
# To use your own boilerplate text use:
# --go-header-file ${SCRIPT_ROOT}/hack/custom-boilerplate.go.txt
|
codegen代码自动生成命令
例子:kube-batch 执行命令代码生成
1
2
3
| GOPATH="project gopath"
${GOPATH}/src/github.com/kubernetes-sigs/kube-batch/hack/nodestat-update-codegen.sh
|
1
2
3
4
5
6
7
| github.com/kubernetes-sigs/kube-batch]# /home/wangb/go_projects/kubebatch/src/github.com/kubernetes-sigs/kube-batch/hack/nodestat-update-codegen.sh
Generating deepcopy funcs
Generating clientset for example.com:v1beta1 at github.com/kubernetes-sigs/kube-batch/pkg/nodestat_client/clientset
Generating listers for example.com:v1beta1 at github.com/kubernetes-sigs/kube-batch/pkg/nodestat_client/listers
Generating informers for example.com:v1beta1 at github.com/kubernetes-sigs/kube-batch/pkg/nodestat_client/informers
|
问题
使用自动生成后的代码,操作自定义CRD时,会报错,如下
1
2
| E0420 11:40:15.739772 8308 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "NodeStat" is registered for version "example.com/v1beta1" in scheme "github.com/kubernetes-sigs/kube-batch/pkg/nodecache_client/clientset/versioned/scheme/register.go:30"
E0420 11:40:15.851896 8308 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: no kind "NodeStat" is registered for version "example.com/v1beta1" in scheme "github.com/kubernetes-sigs/kube-batch/pkg/nodecache_client/clientset/versioned/scheme/register.go:30"
|
手动修改生成代码解决,自定义serializer.DirectCodecFactory
nodestat_client/clientset/versioned/typed/example.com/v1beta1/example.com_client.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
| package v1beta1
import (
v1beta1 "github.com/kubernetes-sigs/kube-batch/pkg/apis/example.com/v1beta1"
"github.com/kubernetes-sigs/kube-batch/pkg/nodestat_client/clientset/versioned/scheme"
"k8s.io/apimachinery/pkg/runtime/serializer"
rest "k8s.io/client-go/rest"
)
func setConfigDefaults(config *rest.Config) error {
gv := v1beta1.SchemeGroupVersion
config.GroupVersion = &gv
config.APIPath = "/apis"
// modify by binge
//config.NegotiatedSerializer = scheme.Codecs.WithoutConversion()
config.NegotiatedSerializer = serializer.DirectCodecFactory{
CodecFactory: scheme.Codecs,
}
if config.UserAgent == "" {
config.UserAgent = rest.DefaultKubernetesUserAgent()
}
return nil
}
|
参考资料