使用 Python 将点分隔值转换为 Go 结构

小伙伴们有没有觉得学习Golang很有意思？有意思就对了！今天就给大家带来《使用 Python 将点分隔值转换为 Go 结构》，以下内容将会涉及到，若是在学习中对其中部分知识点有疑问，或许看了本文就能帮到你！

这是对可以更改配置的应用程序的特定要求（特别是 wso2 identity server，因为我正在使用 go 为其编写 kubernetes 运算符）。但这里确实不相关。我想创建一个解决方案，允许轻松管理大量配置映射以生成 go 结构。这些配置映射在 .csv 中

链接到 .csv - my_configs.csv

我想要， 编写一个自动生成 go 结构的 python 脚本，这样对应用程序配置的任何更改都可以通过简单地执行 python 脚本创建相应的 go 结构来更新。我指的是应用程序本身的配置。例如，可以更改 csv 中的 toml 键名称/可以添加新值。

到目前为止，我已经成功创建了一个 python 脚本，几乎实现了我的目标。脚本是，

import pandas as pd

def convert_to_dict(data):
    result = {}
    for row in data:
        current_dict = result
        for item in row[:-1]:
            if item is not none:
                if item not in current_dict:
                    current_dict[item] = {}
                current_dict = current_dict[item]
    return result

def extract_json_key(yaml_key):
    if isinstance(yaml_key, str) and '.' in yaml_key:
        return yaml_key.split('.')[-1]
    else:
        return yaml_key

def add_fields_to_struct(struct_string,go_var,go_type,json_key,toml_key):
    struct_string += str(go_var) + " " + str(go_type) + ' `json:"' + str(json_key) + ',omitempty" toml:"' +str(toml_key) + '"` ' + "\n"
    return struct_string

def generate_go_struct(struct_name, struct_data):
    struct_name="configurations" if struct_name == "" else struct_name
    struct_string = "type " + struct_name + " struct {\n"
    yaml_key=df['yaml_key'].str.split('.').str[-1]
    
    # base case: generate fields for the current struct level    
    for key, value in struct_data.items():
        selected_rows = df[yaml_key == key]

        if len(selected_rows) > 1:
            go_var = selected_rows['go_var'].values[1]
            toml_key = selected_rows['toml_key'].values[1]
            go_type=selected_rows['go_type'].values[1]
            json_key=selected_rows['json_key'].values[1]
        else:
            go_var = selected_rows['go_var'].values[0]
            toml_key = selected_rows['toml_key'].values[0]
            go_type=selected_rows['go_type'].values[0]
            json_key=selected_rows['json_key'].values[0]

        # add fields to the body of the struct
        struct_string=add_fields_to_struct(struct_string,go_var,go_type,json_key,toml_key)   

    struct_string += "}\n\n"
    
    # recursive case: generate struct definitions for nested structs
    for key, value in struct_data.items():
        selected_rows = df[yaml_key == key]

        if len(selected_rows) > 1:
            go_var = selected_rows['go_var'].values[1]
        else:
            go_var = selected_rows['go_var'].values[0]

        if isinstance(value, dict) and any(isinstance(v, dict) for v in value.values()):
            nested_struct_name = go_var
            nested_struct_data = value
            struct_string += generate_go_struct(nested_struct_name, nested_struct_data)
    
    return struct_string

# read excel
csv_file = "~/downloads/my_configs.csv"
df = pd.read_csv(csv_file)

# remove rows where all columns are nan
df = df.dropna(how='all')
# create the 'json_key' column using the custom function
df['json_key'] = df['yaml_key'].apply(extract_json_key)

data=df['yaml_key'].values.tolist() # read the 'yaml_key' column
data = pd.dataframe({'column':data}) # convert to dataframe

data=data['column'].str.split('.', expand=true) # split by '.'

nested_list = data.values.tolist() # convert to nested list
data=nested_list 

result_json = convert_to_dict(data) # convert to dict (json)

# the generated co code
go_struct = generate_go_struct("", result_json)

# write to file
file_path = "output.go"
with open(file_path, "w") as file:
    file.write(go_struct)

问题是（查看 csv 的下面部分），

authentication.authenticator.basic
authentication.authenticator.basic.parameters
authentication.authenticator.basic.parameters.showAuthFailureReason
authentication.authenticator.basic.parameters.showAuthFailureReasonOnLoginPage
authentication.authenticator.totp
authentication.authenticator.totp.parameters
authentication.authenticator.totp.parameters.showAuthFailureReason
authentication.authenticator.totp.parameters.showAuthFailureReasonOnLoginPage
authentication.authenticator.totp.parameters.encodingMethod
authentication.authenticator.totp.parameters.timeStepSize

这里，由于 basic 和 totp 字段 parameters 重复，因此脚本会混淆自身并生成两个 totpparameters 结构。预期结果是具有 basicparameters 和 totpparameters 结构。 csv 的 yaml_key 列中存在许多类似的重复单词。

我知道这与 go_var = selected_rows['go_var'].values[1] 中索引被硬编码为 1 有关，但很难修复此问题。

有人可以指点我一个答案吗？我认为，

1. 递归函数的问题
2. 生成 json 的代码存在问题 可能是此问题的根本原因。

谢谢！

我也尝试过使用 chatgpt，但是由于这与嵌套和递归有关，因此 chatgpt 提供的答案不是很有效。

更新

我发现包含 properties、pooloptions、endpoint 和 parameters 字段的行存在问题。这是因为它们在 yaml_key 列中重复。

正确答案

我能够解决这个问题。但是，我必须完全使用一种新方法来解决问题，即使用树数据结构，然后遍历它。这是其背后的主要逻辑 - https://www.geeksforgeeks.org/level-顺序树遍历/

这是工作的python代码。

import pandas as pd
from collections import deque

structs=[]
class TreeNode:
    def __init__(self, name):
        self.name = name
        self.children = []
        self.path=""

    def add_child(self, child):
        self.children.append(child)

def create_tree(data):
    root = TreeNode('')
    for item in data:
        node = root
        for name in item.split('.'):
            existing_child = next((child for child in node.children if child.name == name), None)
            if existing_child:
                node = existing_child
            else:
                new_child = TreeNode(name)
                node.add_child(new_child)
                node = new_child
    return root

def generate_go_struct(struct_data):
    struct_name = struct_data['struct_name']
    fields = struct_data['fields']
    
    go_struct = f"type {struct_name} struct {{\n"

    for field in fields:
        field_name = field['name']
        field_type = field['type']
        field_default_val = str(field['default_val'])
        json_key=field['json_key']
        toml_key=field['toml_key']

        tail_part=f"\t{field_name} {field_type} `json:\"{json_key},omitempty\" toml:\"{toml_key}\"`\n\n"

        if pd.isna(field['default_val']):
            go_struct += tail_part
        else:
            field_default_val = "\t// +kubebuilder:default:=" + field_default_val
            go_struct += field_default_val + "\n" + tail_part

    go_struct += "}\n\n"
    return go_struct

def write_go_file(go_structs, file_path):
    with open(file_path, 'w') as file:
        for go_struct in go_structs:
            file.write(go_struct)

def create_new_struct(struct_name):
    struct_name = "Configurations" if struct_name == "" else struct_name
    struct_dict = {
        "struct_name": struct_name,
        "fields": []
    }
    
    return struct_dict

def add_field(struct_dict, field_name, field_type,default_val,json_key, toml_key):
    field_dict = {
        "name": field_name,
        "type": field_type,
        "default_val": default_val,
        "json_key":json_key,
        "toml_key":toml_key
    }
    struct_dict["fields"].append(field_dict)
    
    return struct_dict

def traverse_tree(root):
    queue = deque([root])  
    while queue:
        node = queue.popleft()
        filtered_df = df[df['yaml_key'] == node.path]
        go_var = filtered_df['go_var'].values[0] if not filtered_df.empty else None
        go_type = filtered_df['go_type'].values[0] if not filtered_df.empty else None

        if node.path=="":
            go_type="Configurations"

        # The structs themselves
        current_struct = create_new_struct(go_type)
        
        for child in node.children:  
            if (node.name!=""):
                child.path=node.path+"."+child.name   
            else:
                child.path=child.name

            filtered_df = df[df['yaml_key'] == child.path]
            go_var = filtered_df['go_var'].values[0] if not filtered_df.empty else None
            go_type = filtered_df['go_type'].values[0] if not filtered_df.empty else None
            default_val = filtered_df['default_val'].values[0] if not filtered_df.empty else None

            # Struct fields
            json_key = filtered_df['yaml_key'].values[0].split('.')[-1] if not filtered_df.empty else None
            toml_key = filtered_df['toml_key'].values[0].split('.')[-1] if not filtered_df.empty else None
            
            current_struct = add_field(current_struct, go_var, go_type,default_val,json_key, toml_key)

            if (child.children):
                # Add each child to the queue for processing
                queue.append(child)

        go_struct = generate_go_struct(current_struct)
        # print(go_struct,"\n")        
        structs.append(go_struct)

    write_go_file(structs, "output.go")

csv_file = "~/Downloads/my_configs.csv"
df = pd.read_csv(csv_file) 

sample_data=df['yaml_key'].values.tolist()

# Create the tree
tree = create_tree(sample_data)

# Traverse the tree
traverse_tree(tree)

非常感谢您的帮助！

文中关于的知识介绍，希望对你的学习有所帮助！若是受益匪浅，那就动动鼠标收藏这篇《使用 Python 将点分隔值转换为 Go 结构》文章吧，也可关注golang学习网公众号了解相关技术文章。