python

README

Python Gazelle plugin

This directory contains a plugin for Gazelle that generates BUILD file content for Python code.

Installation

First, you'll need to add Gazelle to your WORKSPACE file. Follow the instructions at https://github.com/bazelbuild/bazel-gazelle#running-gazelle-with-bazel

Next, we need to fetch the third-party Go libraries that the python extension depends on.

Add this to your WORKSPACE:

# To compile the rules_python gazelle extension from source,
# we must fetch some third-party go dependencies that it uses.
load("@rules_python//gazelle:deps.bzl", _py_gazelle_deps = "gazelle_deps")

_py_gazelle_deps()

Next, we'll fetch metadata about your Python dependencies, so that gazelle can determine which package a given import statement comes from. This is provided by the modules_mapping rule. We'll make a target for consuming this modules_mapping, and writing it as a manifest file for Gazelle to read. This is checked into the repo for speed, as it takes some time to calculate in a large monorepo.

Create a file gazelle_python.yaml next to your requirements.txt file. (You can just use touch at this point, it just needs to exist.)

Then put this in your BUILD.bazel file next to the requirements.txt:

load("@pip//:requirements.bzl", "all_whl_requirements")
load("@rules_python//gazelle/manifest:defs.bzl", "gazelle_python_manifest")
load("@rules_python//gazelle/modules_mapping:def.bzl", "modules_mapping")

# This rule fetches the metadata for python packages we depend on. That data is
# required for the gazelle_python_manifest rule to update our manifest file.
modules_mapping(
    name = "modules_map",
    wheels = all_whl_requirements,
)

# Gazelle python extension needs a manifest file mapping from
# an import to the installed package that provides it.
# This macro produces two targets:
# - //:gazelle_python_manifest.update can be used with `bazel run`
#   to recalculate the manifest
# - //:gazelle_python_manifest.test is a test target ensuring that
#   the manifest doesn't need to be updated
gazelle_python_manifest(
    name = "gazelle_python_manifest",
    modules_mapping = ":modules_map",
    # This is what we called our `pip_install` rule, where third-party
    # python libraries are loaded in BUILD files.
    pip_repository_name = "pip",
    # When using pip_parse instead of pip_install, set the following.
    # pip_repository_incremental = True,
    # This should point to wherever we declare our python dependencies
    # (the same as what we passed to the modules_mapping rule in WORKSPACE)
    requirements = "//:requirements_lock.txt",
)

Finally, you create a target that you'll invoke to run the Gazelle tool with the rules_python extension included. This typically goes in your root /BUILD.bazel file:

load("@bazel_gazelle//:def.bzl", "gazelle")
load("@rules_python//gazelle:def.bzl", "GAZELLE_PYTHON_RUNTIME_DEPS")

# Our gazelle target points to the python gazelle binary.
# This is the simple case where we only need one language supported.
# If you also had proto, go, or other gazelle-supported languages,
# you would also need a gazelle_binary rule.
# See https://github.com/bazelbuild/bazel-gazelle/blob/master/extend.rst#example
gazelle(
    name = "gazelle",
    data = GAZELLE_PYTHON_RUNTIME_DEPS,
    gazelle = "@rules_python//gazelle:gazelle_python_binary",
)

That's it, now you can finally run bazel run //:gazelle anytime you edit Python code, and it should update your BUILD files correctly.

A fully-working example is in examples/build_file_generation.

Usage

Gazelle is non-destructive. It will try to leave your edits to BUILD files alone, only making updates to py_* targets. However it will remove dependencies that appear to be unused, so it's a good idea to check in your work before running Gazelle so you can easily revert any changes it made.

The rules_python extension assumes some conventions about your Python code. These are noted below, and might require changes to your existing code.

Note that the gazelle program has multiple commands. At present, only the update command (the default) does anything for Python code.

Directives

You can configure the extension using directives, just like for other languages. These are just comments in the BUILD.bazel file which govern behavior of the extension when processing files under that folder.

See https://github.com/bazelbuild/bazel-gazelle#directives for some general directives that may be useful. In particular, the resolve directive is language-specific and can be used with Python. Examples of these directives in use can be found in the /gazelle/testdata folder in the rules_python repo.

Python-specific directives are as follows:

Directive	Default value
# gazelle:python_extension	enabled
Controls whether the Python extension is enabled or not. Sub-packages inherit this value. Can be either "enabled" or "disabled".
# gazelle:python_root	n/a
Sets a Bazel package as a Python root. This is used on monorepos with multiple Python projects that don't share the top-level of the workspace as the root.
# gazelle:python_manifest_file_name	gazelle_python.yaml
Overrides the default manifest file name.
# gazelle:python_ignore_files	n/a
Controls the files which are ignored from the generated targets.
# gazelle:python_ignore_dependencies	n/a
Controls the ignored dependencies from the generated targets.
# gazelle:python_validate_import_statements	true
Controls whether the Python import statements should be validated. Can be "true" or "false"
# gazelle:python_generation_mode	package
Controls the target generation mode. Can be "package" or "project"
# gazelle:python_library_naming_convention	$package_name$
Controls the py_library naming convention. It interpolates $package_name$ with the Bazel package name. E.g. if the Bazel package name is foo, setting this to $package_name$_my_lib would result in a generated target named foo_my_lib.
# gazelle:python_binary_naming_convention	$package_name$_bin
Controls the py_binary naming convention. Follows the same interpolation rules as python_library_naming_convention.
# gazelle:python_test_naming_convention	$package_name$_test
Controls the py_test naming convention. Follows the same interpolation rules as python_library_naming_convention.
# gazelle:resolve py ...	n/a
Instructs the plugin what target to add as a dependency to satisfy a given import statement. The syntax is # gazelle:resolve py import-string label where import-string is the symbol in the python import statement, and label is the Bazel label that Gazelle should write in deps.

Libraries

Python source files are those ending in .py but not ending in _test.py.

First, we look for the nearest ancestor BUILD file starting from the folder containing the Python source file.

If there is no py_library in this BUILD file, one is created, using the package name as the target's name. This makes it the default target in the package.

Next, all source files are collected into the srcs of the py_library.

Finally, the import statements in the source files are parsed, and dependencies are added to the deps attribute.

Tests

Python test files are those ending in _test.py.

A py_test target is added containing all test files as srcs.

Binaries

When a __main__.py file is encountered, this indicates the entry point of a Python program.

A py_binary target will be created, named [package]_bin.

Developing on the extension

Gazelle extensions are written in Go. Ours is a hybrid, which also spawns a Python interpreter as a subprocess to parse python files.

The Go dependencies are managed by the go.mod file. After changing that file, run go mod tidy to get a go.sum file, then run bazel run //:update_go_deps to convert that to the gazelle/deps.bzl file. The latter is loaded in our /WORKSPACE to define the external repos that we can load Go dependencies from.

Then after editing Go code, run bazel run //:gazelle to generate/update go_* rules in the BUILD.bazel files in our repo.

Documentation

Index

• func NewLanguage() language.Language
• type Configurer
  • func (py *Configurer) CheckFlags(fs *flag.FlagSet, c *config.Config) error
  • func (py *Configurer) Configure(c *config.Config, rel string, f *rule.File)
  • func (py *Configurer) KnownDirectives() []string
  • func (py *Configurer) RegisterFlags(fs *flag.FlagSet, cmd string, c *config.Config)
• type Python
  • func (py *Python) Fix(c *config.Config, f *rule.File)
  • func (py *Python) GenerateRules(args language.GenerateArgs) language.GenerateResult
  • func (*Python) Kinds() map[string]rule.KindInfo
  • func (py *Python) Loads() []rule.LoadInfo
• type Resolver
  • func (py *Resolver) Embeds(r *rule.Rule, from label.Label) []label.Label
  • func (py *Resolver) Imports(c *config.Config, r *rule.Rule, f *rule.File) []resolve.ImportSpec
  • func (*Resolver) Name() string
  • func (py *Resolver) Resolve(c *config.Config, ix *resolve.RuleIndex, rc *repo.RemoteCache, r *rule.Rule, ...)

Functions

func NewLanguage

func NewLanguage() language.Language

NewLanguage initializes a new Python that satisfies the language.Language interface. This is the entrypoint for the extension initialization.

Types

type Configurer

type Configurer struct{}

Configurer satisfies the config.Configurer interface. It's the language-specific configuration extension.

func (*Configurer) CheckFlags

func (py *Configurer) CheckFlags(fs *flag.FlagSet, c *config.Config) error

CheckFlags validates the configuration after command line flags are parsed. This is called once with the root configuration when Gazelle starts. CheckFlags may set default values in flags or make implied changes.

func (*Configurer) Configure

func (py *Configurer) Configure(c *config.Config, rel string, f *rule.File)

Configure modifies the configuration using directives and other information extracted from a build file. Configure is called in each directory.

c is the configuration for the current directory. It starts out as a copy of the configuration for the parent directory.

rel is the slash-separated relative path from the repository root to the current directory. It is "" for the root directory itself.

f is the build file for the current directory or nil if there is no existing build file.

func (*Configurer) KnownDirectives

func (py *Configurer) KnownDirectives() []string

KnownDirectives returns a list of directive keys that this Configurer can interpret. Gazelle prints errors for directives that are not recoginized by any Configurer.

func (*Configurer) RegisterFlags

func (py *Configurer) RegisterFlags(fs *flag.FlagSet, cmd string, c *config.Config)

RegisterFlags registers command-line flags used by the extension. This method is called once with the root configuration when Gazelle starts. RegisterFlags may set an initial values in Config.Exts. When flags are set, they should modify these values.

type Python

type Python struct {
	Configurer
	Resolver
}

Python satisfies the language.Language interface. It is the Gazelle extension for Python rules.

func (*Python) Fix

func (py *Python) Fix(c *config.Config, f *rule.File)

Fix repairs deprecated usage of language-specific rules in f. This is called before the file is indexed. Unless c.ShouldFix is true, fixes that delete or rename rules should not be performed.

func (*Python) GenerateRules

func (py *Python) GenerateRules(args language.GenerateArgs) language.GenerateResult

GenerateRules extracts build metadata from source files in a directory. GenerateRules is called in each directory where an update is requested in depth-first post-order.

func (*Python) Kinds

func (*Python) Kinds() map[string]rule.KindInfo

Kinds returns a map that maps rule names (kinds) and information on how to match and merge attributes that may be found in rules of those kinds.

func (*Python) Loads

func (py *Python) Loads() []rule.LoadInfo

Loads returns .bzl files and symbols they define. Every rule generated by GenerateRules, now or in the past, should be loadable from one of these files.

type Resolver

type Resolver struct{}

Resolver satisfies the resolve.Resolver interface. It resolves dependencies in rules generated by this extension.

func (*Resolver) Embeds

func (py *Resolver) Embeds(r *rule.Rule, from label.Label) []label.Label

Embeds returns a list of labels of rules that the given rule embeds. If a rule is embedded by another importable rule of the same language, only the embedding rule will be indexed. The embedding rule will inherit the imports of the embedded rule.

func (*Resolver) Imports

func (py *Resolver) Imports(c *config.Config, r *rule.Rule, f *rule.File) []resolve.ImportSpec

Imports returns a list of ImportSpecs that can be used to import the rule r. This is used to populate RuleIndex.

If nil is returned, the rule will not be indexed. If any non-nil slice is returned, including an empty slice, the rule will be indexed.

func (*Resolver) Name

func (*Resolver) Name() string

Name returns the name of the language. This is the prefix of the kinds of rules generated. E.g. py_library and py_binary.

func (*Resolver) Resolve

func (py *Resolver) Resolve(
	c *config.Config,
	ix *resolve.RuleIndex,
	rc *repo.RemoteCache,
	r *rule.Rule,
	modulesRaw interface{},
	from label.Label,
)

Resolve translates imported libraries for a given rule into Bazel dependencies. Information about imported libraries is returned for each rule generated by language.GenerateRules in language.GenerateResult.Imports. Resolve generates a "deps" attribute (or the appropriate language-specific equivalent) for each import according to language-specific rules and heuristics.