Closures In Golang And Python

calendar Nov 10, 2022 · 12 min read · golang python English  ·
Share on: linkedin copy

Overview

For people new to Golang and Python, this article explains what closures are as well as sometips that need to know about when using closures in these two programming languages. To make it easier for you to copy down and run on your own server, the code section would provide the package, import, and other repeated parts. The negative aspect is that this results in some redundancy, so please understand if this causes any inconvenience.

What Is Closures?

a closure is a record storing a function together with an environment.

Closure's two factors: function and environment

  1. function：Refers to the fact that when closures are actually implemented, they are often done by calling an external function that returns its internal function. An internal function may be an internal real-name function, an anonymous function, or a lambda expression.
  2. environment：In practice, the referenced environment is the environment of the external function and the closure holds/records all the environment of the external function at the time when it's being generated.

To summarize, a closure is a function that extends the scope by retaining the bindings of free variables (variables not bound in the local scope) that existed when the function was defined, so that when the function is called, the definition scope is no longer available, but those bindings can still be used. Here is a look at the common usage of closures:

 1// golang version
 2package main
 3
 4import "fmt"
 5
 6func outer() func(v int) {
 7    x := 1
 8    return func(v int) {
 9        y := x + v
10        fmt.Println(x)
11        fmt.Println(y)
12    }
13}
14
15func main() {
16    a := outer()
17    a(1)
18    a(2)
19}
20
21// result
221
232
241
253

In the logic inside the golang example, "a" is a closure, the closure function is the internal func(v int){}, the closure environment is the external "x", since the external environment is "captured", so each execution of the closure "x" is 1, and the final output result is 1,2,1,3.

 1# python version
 2def outer():
 3    x = 1
 4
 5    def inner(v):
 6        y = x + v
 7        print(x)
 8        print(y)
 9    return inner
10
11a = outer()
12a(1)
13a(2)
14
15# result
161
172
181
193

When comes to Python example, the same closure function is inner(v), the closure environment is "x", so each time the closure is executed "x" is 1 (because the closure environment is captured), and then the logic inside the closure is to add "x" to the parameters passed in by the closure, so the final output is 1,2,1,3.

Closures In Golang

Anonymous Function

Since anonymous functions are used very frequently in golang, let's start with this: Anonymous functions and the free variables they "capture" are called closures, and they are closed whether used normally, in a for loop, or in a defer:

 1package main
 2
 3import "fmt"
 4
 5func main() {
 6    x := 1
 7    a := func(v int) {
 8            y := x + v
 9            fmt.Println(x)
10            fmt.Println(y)
11    }
12    a(2)
13    a(3)
14}
15
16// result
171
183
191
204

In the above example a is a closure, the closure function is the anonymous function func(v int){}, the closure environment is x, so the result is 1,3,1,4.

 1package main
 2
 3import "fmt"
 4
 5func main() {
 6    y := 2
 7
 8    defer func() {
 9        fmt.Println(y)
10    }()
11}
12
13// result
142

The same goes for the anonymous function in defer, which is func(){} in defer, and the closure environment is "y", so the output is 2.

Modifies The Closure Environment

First of all, all reference passing in golang is a value passing, if there is something like reference passing (which will also be mentioned later in this article for convenience), it is actually the "value" of the underlying pointer that is passed, thus achieving the so-called reference passing.

  1. If you want to modify the closure environment inside the closure (in the closure function), golang is easy to do that. Owing to the fact that golang is a declarative language, assignment and declaration are written differently(:= for declaration, = for assignment); and golang closure "captures" the essence of the closure environment is reference passing rather than value passing, so directly modify it is ok, see the following example:
 1package main
 2
 3import "fmt"
 4
 5func make_avg() func(v int) {
 6        count := 0
 7        total := 0
 8        return func(v int) {
 9                count += 1
10                total += v
11                fmt.Println(float32(total)/float32(count))
12        }
13}
14
15func main() {
16        a := make_avg()
17        a(1)
18        a(2)
19        a(3)
20}
21
22// result
231
241.5
252

The example is to calculate the average value, the closure is "a", the closure function is the internal func(v int){} anonymous function, the closure environment is "count" and "total"; count += 1, total += v are direct modifications to the behavior of the closure environment and get the desired effect.

Specifically, you can use golang closures to "catch" the essence of the closure environment is reference passing this feature inside the anonymous function (inside the closure function) modify the global variables (closure environment), see the following example:

 1package main
 2
 3import (
 4    "fmt"
 5)
 6
 7var x int = 1
 8
 9func main() {
10    a := func() {
11        x += 1
12    }
13    fmt.Println(x)
14    a()
15    fmt.Println(x)
16}
17
18// result
191
202
  1. Modify the closure environment outside the closure

You must have some questions, "outside the closure" can modify the closure environment? In fact, it is possible in golang, remember the following two sentences:

  • If all the variables of the external function are local, that is, the life cycle ends when the external function ends, then the environment of the closure is also closed.
  • Conversely, then the closure is no longer closed, and changes to globally visible variables will have an effect on the variables within the closure.

Generally speaking, if the environment of a closure can be modified by a pointer, then the environment of the closure can be modified from outside the closure, see the following example:

 1package main
 2
 3import "fmt"
 4
 5func foo1(x *int) func() {
 6    return func() {
 7        *x = *x + 1
 8        fmt.Println(*x)
 9    }
10}
11func foo2(x int) func() {
12    return func() {
13        x = x + 1
14        fmt.Println(x)
15    }
16}
17
18func main() {
19    x := 133
20    f1 := foo1(&x)
21    f2 := foo2(x)
22    f1()
23    f1()
24    f2()
25    f2()
26
27    x = 233
28    f1()
29    f1()
30    f2()
31    f2()
32
33    foo1(&x)()
34    foo1(&x)()
35    foo2(x)()
36    foo2(x)()
37}

The logic inside the two closures needs to be analyzed, one is the sum of pointer variables, according to what we said earlier, "the closure environment can be modified by pointers", each time the closure is executed or directly assigned outside will really change the value of the variable, while the "foo2", which does not use pointers, is the normal closure, that is, the closure environment is only inside the closure; so the first four groups of output as follows:

1134
2135
3134
4135

The middle four groups are being accumulated by the "f1" closure on the modified 233 because the value of "x" is forced externally, and the "f2" closure is being accumulated in its own environment, so the output is:

1234
2235
3136
4137

The last four groups generate four new closures, so the "foo1" part is still cumulative based on the current "x" value, and the cumulative value actually acts in the global variable "x"; the cumulative in foo2 is still inside its own closure, so the output is:

1236
2237
3238
4238

With this example we distinguish between modifying the closure environment from inside the closure and from outside the closure.

Delayed Binding Of Closures

This problem is every golang newbie will encounter, very puzzling problem; please remember the following sentence: when executing the closure, the closure environment declaration cycle is guaranteed, and will go to the external environment to find the latest closure environment (value), the following example when executing the closure "i" is the closure environment, when executing the closure the latest value is already 10, so all will output 10.

 1package main
 2
 3import "fmt"
 4
 5func main() {
 6    var handlers []func()
 7    for i := 0; i < 10; i++ {
 8        handlers = append(handlers, func() {
 9                fmt.Println(i)
10        })
11    }
12    for _, handler := range handlers {
13        handler()
14    }
15}
16
17// result
1810
1910
2010
2110
2210
2310
2410
2510
2610
2710

The solution is to copy an environment variable in the for loop that is not referenced by the closure, and then use that value instead of the closure environment, with the following modified version:

 1package main
 2
 3import "fmt"
 4
 5func main() {
 6   var handlers []func()
 7   for i := 0; i < 10; i++ {
 8      // a is not a closure environment because it is redeclared each time
 9      a := i
10      handlers = append(handlers, func() {
11              fmt.Println(a)
12      })
13   }
14   for _, handler := range handlers {
15      handler()
16   }
17}
18
19// result
200
211
222
233
244
255
266
277
288
299

In fact, the principle is clear, it is not about the for loop, normal use, defer use will be bound by this principle, the execution of the closure, the closure environment declaration cycle is guaranteed, and will go to the external environment to find the latest closure environment (value)

 1package main
 2
 3import "fmt"
 4
 5func main() {
 6    x, y := 1, 2
 7
 8    defer func(a int) {
 9        fmt.Println(a, y)
10    }(x)
11
12    x += 100
13    y += 100
14}
15
16// The output, y, is the closure environment, so the execution of the closure will go to the latest value, while a is not the closure environment, copying the value of x so it is not implicated
171 102

The use of anonymous functions in go routines is a common scenario and suffers from this problem, see the following example:

 1package main
 2
 3import (
 4    "fmt"
 5    "time"
 6)
 7
 8func show(val int) {
 9    fmt.Println(val)
10}
11
12func main() {
13    values := []int{1, 2, 3, 5}
14    for _, val := range values {
15        go show(val)
16    }
17    time.Sleep(time.Second)
18}
19
20// The four outputs 1,2,3,5 will be output each time, although in a different order, because no anonymous function is used, and it is not a closure
215
221
233
242

The four outputs 1,2,3,5 will be output each time, although in a different order, because no anonymous function is used, and it is not a closure.

 1package main
 2
 3import (
 4        "fmt"
 5        "time"
 6)
 7
 8func main() {
 9    values := []int{1, 2, 3, 5}
10    for _, val := range values {
11        go func(){
12            fmt.Println(val)
13        }()
14    }
15    time.Sleep(time.Second)
16}
17
18// output
195
205
215
225

The modification method is the same as inside the for loop example, using passing variables to avoid the closure environment.

 1package main
 2
 3import (
 4        "fmt"
 5        "time"
 6)
 7
 8func main() {
 9    values := []int{1, 2, 3, 5}
10    for _, val := range values {
11        go func(val int){
12            fmt.Println(val)
13        }(val)
14    }
15    time.Sleep(time.Second)
16}
17
18// output
191
205
213
222

Closures In Python

Modifies The Closure Environment

  1. Modifying the closure environment from within

Since python is not a declarative language, a "=" eats all, we need to use the nonlocal parameter to explicitly declare the variable as a closure environment, and not a local variable, look at the following example, if the same way as golang directly change, sometimes there will be a problem.

use Python list as a closure environment

 1def make_avg(count, total):
 2    count = []
 3
 4    def avg(v):
 5        count.append(v)
 6        total = sum(count)
 7        print(sum(count)/len(count))
 8    return avg
 9
10
11a = make_avg(0, 0)
12a(1)
13a(2)
14a(3)
15
16
17# output
181.0
191.5
202.0

use Python character as a closure environment

 1def make_avg(count, total):
 2    count = 0
 3    total = 0
 4
 5    def avg(v):
 6        count += 1
 7        total += v
 8        print(total/count)
 9    return avg
10
11
12a = make_avg(0, 0)
13a(1)
14a(2)
15a(3)
16
17
18# error
19Traceback (most recent call last):
20  File "/root/code/linux/blog/aa.py", line 14, in <module>
21    a(1)
22  File "/root/code/linux/blog/aa.py", line 7, in avg
23    count += 1
24UnboundLocalError: local variable 'count' referenced before assignment

That is because python has two types of data, one for mutable data types, one for immutable data types, when passing variable data types use reference passing, this and golang closure directly modified characteristics match, but immutable data types such as the above string will be a problem, because python will take him as a newly generated local variables, so the solution is to use nonlocal to tell the python interpreter, This variable is for the closure environment, so that it can run properly, the correct way to write the following:

 1def make_avg(count, total):
 2    count = 0
 3    total = 0
 4
 5    def avg(v):
 6        nonlocal count, total
 7        count += 1
 8        total += v
 9        print(total/count)
10    return avg
11
12
13a = make_avg(0, 0)
14a(1)
15a(2)
16a(3)
17
18# output
191.0
201.5
212.0

Closures And Decorators

I was going to talk about python decorators, but I decided to leave it for another time because there is already a lot of space here.

Summary

There are many examples above, after reading each example, to understand why, then the closure problem will be solved;

Keep in mind, cuz the closure is not commonly used, and it is not very practical, you don't have to use closures if the work is not necessary to use. Or may trigger some memory leaks, which is not a small problem.

  1. two main elements of the closure: function and environment
  2. closure is a function that extends the scope, it will retain the bindings of free variables (variables not bound in the local scope) that existed when the function was defined, so when the function is called, so the definition scope is not available, but can still use those bindings
  3. the use of anonymous functions in golang is actually a closure
  4. the closure environment variables can be modified from outside the closure by pointers
  5. golang closures delayed binding problem: when executing a closure, the declaration cycle of the closure environment is guaranteed, and it will go to the external environment to find the latest closure environment (value)
  6. python closures use nonlocal to declare variables as closure environment, instead of local variables

The above analysis is my understanding of the closure, I hope golang, python newcomers can avoid a must-step pit after reading, if you have questions about the environment feel free to comment or email contact, thank you.

Reference Links

https://juejin.cn/post/6844904133208604679
https://www.jianshu.com/p/fa21e6fada70
https://zhuanlan.zhihu.com/p/92634505