Click to follow me on hashnode and read all my upcoming stories!

To get started, you first need to import the Google UUID library into your Golang project.

go get -u github.com/google/uuid

Once you have imported the library, you could use the uuid.New() method to generate a random UUID. This method takes no arguments and is the easiest one to get started. 🆒

The challenge with using this is how to create a Unit Test for the code 👊. In the Unit Test, we expect to get a constant known behaviour from our code. But, the behaviour of uuid.New() method is to generate random sequences on each execution.

Let me show you what I mean by using a small code. Below, I am writing a very simple test. I will run this test multiple times and we will check the output. All the generated UUIDs for each different run will be random.

func Test_UUIDGeneration(t *testing.T) {
 uuid := uuid.New()

 fmt.Println(fmt.Sprintf("UUID: %s", uuid))
}

------------ OUTPUT ---------------

UUID: c26a1010-a207-47cb-b399-9de8acad3bba
UUID: e4c1776f-cd94-4710-b5eb-710c081df916
UUID: 923c129c-ba0b-4534-aadf-db98470fd99c

Note: If you run the above test locally with go test make sure to disable the cache. Use--count=1 in the command to disable caching otherwise the test will generate same uuids

Consider you are writing a service layer that generates the UUID and persist it in the DB. You write a Unit Test but cannot assert the value of the generated UUID. Because, on every new execution on the CI, the value will be different. How will you test the code and avoid randomness?

Here is a small code to mimic a service layer and the test:

type Employee struct {
 ID        uuid.UUID `json:"id"`
 FirstName string    `json:"first_name"`
 LastName  string    `json:"last_name"`
}

// Storage interface is used to persist employee to DB
type Storage interface {
 Save(employee Employee) error
}

// Employee service orchestrate the creation of employee
type EmployeeService struct {
 storage Storage
}

// For simplicity the layer only calls the storage layer to persist the value.
// However, in a real world the service layer will do more than this.
func (e *EmployeeService) Create(firstName, lastName string) error {
 employee := Employee{
  ID:        uuid.New(),
  FirstName: firstName,
  LastName:  lastName,
 }

 return e.storage.Save(employee)
}

type mockStorage struct {
 mock.Mock
}

// We mock the storage layer to see if the value and type is correctly passed
// to the method
func (m *mockStorage) Save(employee Employee) error {
 args := m.Called(employee)
 return args.Error(0)
}

func TestEmployeeService_Create(t *testing.T) {
 t.Parallel()

 expectedEmployee := Employee{
  ID:        uuid.New(), // This will always generate new random uuid
  FirstName: "John",
  LastName:  "Doe",
 }

 storage := &mockStorage{}
 employeeService := EmployeeService{storage: storage}

 storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  employee, ok := x.(Employee)

  if !ok {
   return false
  }

  return assert.EqualValues(t, expectedEmployee, employee)
 })).Return(nil)

 err := employeeService.Create("John", "Doe")

 assert.NoError(t, err)
 storage.AssertExpectations(t)
}

If I run the above test it will fail 🔴 :

=== RUN   TestEmployeeService_Create
=== PAUSE TestEmployeeService_Create
=== CONT  TestEmployeeService_Create
/tests/data_test.go:89:
      Error Trace:    /tests/data_test.go:89
                      /tests/value.go:586
                      /tests/value.go:370
                      /tests/data_test.go:66
                      /tests/data_test.go:58
                      /tests/data_test.go:92
      Error:          Not equal:
                      expected: tests.Employee{ID:uuid.UUID{0xeb, 0x5, 0x6f, 0x11, 0x47, 0x8, 0x42, 0x2d, 0xb5, 0x23, 0x47, 0x2, 0x40, 0x18, 0x85, 0x94}, FirstName:"John", LastName:"Doe"}
                      actual  : tests.Employee{ID:uuid.UUID{0xb4, 0xb7, 0x8f, 0x21, 0x36, 0xc8, 0x4e, 0x3e, 0x9a, 0xa3, 0x96, 0x79, 0xd3, 0xcc, 0xbc, 0xdc}, FirstName:"John", LastName:"Doe"}

                      Diff:
                      --- Expected
                      +++ Actual
                      @@ -2,3 +2,3 @@
                        ID: (uuid.UUID) (len=16) {
                      -  00000000  eb 05 6f 11 47 08 42 2d  b5 23 47 02 40 18 85 94  |..o.G.B-.#G.@...|
                      +  00000000  b4 b7 8f 21 36 c8 4e 3e  9a a3 96 79 d3 cc bc dc  |...!6.N>...y....|
                        },
      Test:           TestEmployeeService_Create

You could see that the UUIDs were generated by the service layer in the logs. To use assert.EqualValues or assert.Equals on a struct, all the attribute values should match. The UUID in the expected employee instance is different. assert.Equals would need the UUIDs that are known to the test when you write it, randomness here is the big problem.

What would you do to avoid this?

Many people to avoid the problem will think of not writing the test 😅. Let’s see the different ways to overcome this problem. But please DO NOT AVOID WRITING TESTS!!!! 😃

1. Manually check all the struct attributes and avoid the UUID check

Our assumption is that this library is open-source and well-maintained by the community. We can rely on the fact that things will work fine. We skip the UUID check part but check other attributes eg:

storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  employee, ok := x.(Employee)

  if !ok {
   return false
  }

  return employee.ID != uuid.Nil && employee.FirstName == expectedEmployee.FirstName && employee.LastName == expectedEmployee.LastName
 })).Return(nil)

This is ok if you have a struct with few attributes. But, this will become a headache if you have 15–20 attributes. This indeed is not a scalable approach.

2. Parse the UUID To Check For Error

Parse the UUID to check if the service generates the UUID and if there is no issue. For other attributes do exactly what we did in the previous example. This will also suffer from the same problem as the previous solution

_, err := uuid.Parse(employee.ID.String())

return err == nil && employee.FirstName == expectedEmployee.FirstName && employee.LastName == expectedEmployee.LastName

3. Pass a UUID Generator Function Type

A separate uuid generator function is passed to the struct to generate the uuid. A test generator is then used in the unit test to mock and pass a static UUID. An assertion on this static value will verify the behaviour of the code. Eg:

type UUIDGenerator func() uuid.UUID

var DefaultUUIDGenerator = func() uuid.UUID {
 return uuid.New()
}

type EmployeeService struct {
 storage       Storage
 uuidGenerator UUIDGenerator
}

func (e *EmployeeService) Create(firstName, lastName string) error {
 employee := Employee{
  ID:        e.uuidGenerator(),
  FirstName: firstName,
  LastName:  lastName,
 }

 return e.storage.Save(employee)
}

func TestEmployeeService_Create(t *testing.T) {
 t.Parallel()

 expectedID := uuid.New()

 expectedEmployee := Employee{
  ID:        expectedID,
  FirstName: "John",
  LastName:  "Doe",
 }

 storage := &mockStorage{}

 uuidGenerator := func() uuid.UUID {
  return expectedID
 }

 employeeService := EmployeeService{storage: storage, uuidGenerator: uuidGenerator}

 storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  employee, ok := x.(Employee)

  if !ok {
   return false
  }

  return assert.EqualValues(t, expectedEmployee, employee)
 })).Return(nil)

 err := employeeService.Create("John", "Doe")

 assert.NoError(t, err)
 storage.AssertExpectations(t)
}

This solution is also good. It will allow us to assert the entire entity and don’t have to check every field like in the previous two examples. This may become a problem if you have to use it at many places in your code base.

One could also add an interface to generate the UUID and mock the interface in the test to provide the static UUID. This is also an alternative to the function type and is acceptable.

Both the function type and interface will help to reuse and it will work fine. But what if I tell you there is a secret way and you don’t have to do all these shenanigans? 👇

4. Use SetRand 🚀

Let’s look at the documentation of the library. There is a way to generate deterministic UUID, using uuid.SetRand() . If we use the correct Seed Source to theSetRand function it will allow us to get deterministic UUIDs.

Let’s see how:

func Test_uuidTest(t *testing.T) {
 uuid.SetRand(rand.New(rand.NewSource(1)))

 val1 := uuid.New()
 val2 := uuid.New()

 assert.EqualValues(t, uuid.MustParse("52fdfc07-2182-454f-963f-5f0f9a621d72"), val1, fmt.Sprintf("generated %v", val1))
 assert.EqualValues(t, uuid.MustParse("9566c74d-1003-4c4d-bbbb-0407d1e2c649"), val2, fmt.Sprintf("generated %v", val2))
}

I add another test above to show how to use the SetRand method. I pass a new *rand.Rand instance by setting the seed value to the rand.NewSource(1) . When you do this the UUIDs generated will be deterministic in nature. They will be exactly the same on each run irrespective of the platform you run on.

To verify this, I ran the same example on the go playground here. You will see below that the generated UUIDs match our assertion value.

So let’s update our existing test to see how to use this new trick.

func TestEmployeeService_Create(t *testing.T) {
 uuid.SetRand(rand.New(rand.NewSource(1)))

 expectedEmployee := Employee{
  ID:        uuid.MustParse("52fdfc07-2182-454f-963f-5f0f9a621d72"),
  FirstName: "John",
  LastName:  "Doe",
 }

 storage := &mockStorage{}

 employeeService := EmployeeService{storage: storage}

 storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  employee, ok := x.(Employee)

  if !ok {
   return false
  }

  return assert.EqualValues(t, expectedEmployee, employee)
 })).Return(nil)

 err := employeeService.Create("John", "Doe")

 assert.NoError(t, err)
 storage.AssertExpectations(t)
}

There is one caveat with this approach. The math/rand source that we have used is not thread-safe. As per the documentation:

// NewSource returns a new pseudo-random Source seeded with the given value.
// Unlike the default Source used by top-level functions, this source is not
// safe for concurrent use by multiple goroutines.
// The returned Source implements Source64.

So having multiple tests running parallel would cause a data race issue. I tried to overcome this but there’s no straightforward way. I don’t want to add another layer of complexity to avoid the first one 😅 it’s not productive at all.

I looked into how these libraries did the unit test. I realised that both google/uuid and golang math/rand package do not use t.Parallel() for the tests. So, The simplest way to fix the issue would be to remove t.Parallel() from all such unit tests ✅.

5. Use google/go-cmp

After publishing this post, my awesome readers reached out to me with one more approach. I am really thankful for all the feedback. The google/go-cmp library allows to skip/ignore fields from comparison.

There are two kinds of people, one who wants to compare all the fields of the struct and some would prefer to ignore the UUID check because the google/uuid library is battle tested. So based on which category you belong to you could use the previous approach or you could use this approach.

Let’s rewrite our existing test to use the cmpopts.IgnoreFields

func TestEmployeeService_Create(t *testing.T) {
 expectedEmployee := Employee{
  FirstName: "John",
  LastName:  "Doe",
 }

 storage := &mockStorage{}

 employeeService := EmployeeService{storage: storage}

 storage.On("Save", mock.MatchedBy(func(x interface{}) bool {
  actual, ok := x.(Employee)

  if !ok {
   return false
  }

  var diff string
  if diff = cmp.Diff(expectedEmployee, actual, cmpopts.IgnoreFields(Employee{}, "ID")); diff != "" {
   t.Errorf("Save() mismatch (-want +got):\n%s", diff)
  }

  return diff == ""
 })).Return(nil)

 err := employeeService.Create("John", "Doe")

 assert.NoError(t, err)
 storage.AssertExpectations(t)
}

On failure ☠️ the test will show a diff comparison of the fields that didn’t match like below

=== RUN   TestEmployeeService_Create
    /Users/neenadingole/codes/opensource/gotest-ls/main_test.go:275: Save() mismatch (-want +got):
          main.Employee{
                ... // 1 ignored field
        -       FirstName: "Face",
        +       FirstName: "John",
                LastName:  "Doe",
          }
    /Users/neenadingole/codes/opensource/gotest-ls/main_test.go:275: Save() mismatch (-want +got):
          main.Employee{
                ... // 1 ignored field
        -       FirstName: "Face",
        +       FirstName: "John",
                LastName:  "Doe",
          }
--- FAIL: TestEmployeeService_Create (0.00s)
panic:

mock: Unexpected Method Call
-----------------------------

Save(main.Employee)
                0: main.Employee{ID:uuid.UUID{0x4d, 0x8e, 0x29, 0x66, 0x29, 0xdd, 0x4b, 0xbe, 0xa7, 0x30, 0x34, 0xa0, 0x7a, 0xe7, 0xb3, 0x94}, FirstName:"John", LastName:"Doe"}

The closest call I have is:

Save(mock.argumentMatcher)
                0: mock.argumentMatcher{fn:reflect.Value{typ:(*reflect.rtype)(0x12fd9e0), ptr:(unsafe.Pointer)(0xc00010ce40), flag:0x13}}


Diff: 0: FAIL:  (main.Employee={4d8e2966-29dd-4bbe-a730-34a07ae7b394 John Doe}) not matched by func(interface {}) bool [recovered]
        panic:

It becomes easy to navigate the test and fix the issue and you could ignore multiple fields from the struct comparison.

I am going to use this library in my projects. This is also a great learning for me by writing and sharing this blog post.

Conclusion

I hope this helps you to write better tests for your code when using the google uuid package. I am very keen on testing code and finding ways to ease testing for developers. Having proper tests helps the team move faster. It also provides a safety net to avoid accidental bug leaks or any dev mistakes.

There are different pros and cons to each of the methods discussed. It will depend on what works for you and what doesn’t. For a small codebase, I don’t like using the generator function approach. I would use the SetRand for such cases. For larger struct or ignoring the field comparison for some then I would also prefer the go-cmp library which does a really good job.

I was curious to see if there is any simple way for unit testing. This post is my research on this topic, but I know there may be some approaches I missed. So, I would like to hear those from others. I will definitely be happy to update my findings with your thoughts 🙏.

Thank you so much for taking the time to read my blog and sharing it with others. Your ❤️ support means a lot to me, and I truly appreciate it. It’s people like you who inspire me to keep writing and sharing my thoughts with the world. Thank you again for your kindness and support.

Supporting Information

A Sample Test From google/uuid package:

// No t.Parallel() used
func TestSetRand(t *testing.T) {
 myString := "805-9dd6-1a877cb526c678e71d38-7122-44c0-9b7c-04e7001cc78783ac3e82-47a3-4cc3-9951-13f3339d88088f5d685a-11f7-4078-ada9-de44ad2daeb7"

 SetRand(strings.NewReader(myString))
 uuid1 := New()
 uuid2 := New()

 SetRand(strings.NewReader(myString))
 uuid3 := New()
 uuid4 := New()

 if uuid1 != uuid3 {
  t.Errorf("expected duplicates, got %q and %q", uuid1, uuid3)
 }
 if uuid2 != uuid4 {
  t.Errorf("expected duplicates, got %q and %q", uuid2, uuid4)
 }
}

A sample test from golang math/rand package:

// No t.Parallel() used
func testReadUniformity(t *testing.T, n int, seed int64) {
 r := New(NewSource(seed))
 buf := make([]byte, n)
 nRead, err := r.Read(buf)
 if err != nil {
  t.Errorf("Read err %v", err)
 }
 if nRead != n {
  t.Errorf("Read returned unexpected n; %d != %d", nRead, n)
 }

 // Expect a uniform distribution of byte values, which lie in [0, 255].
 var (
  mean       = 255.0 / 2
  stddev     = 256.0 / math.Sqrt(12.0)
  errorScale = stddev / math.Sqrt(float64(n))
 )

 expected := &statsResults{mean, stddev, 0.10 * errorScale, 0.08 * errorScale}

 // Cast bytes as floats to use the common distribution-validity checks.
 samples := make([]float64, n)
 for i, val := range buf {
  samples[i] = float64(val)
 }
 // Make sure that the entire set matches the expected distribution.
 checkSampleDistribution(t, samples, expected)
}

// No t.Parallel() used
func TestReadUniformity(t *testing.T) {
 testBufferSizes := []int{
  2, 4, 7, 64, 1024, 1 << 16, 1 << 20,
 }
 for _, seed := range testSeeds {
  for _, n := range testBufferSizes {
   testReadUniformity(t, n, seed)
  }
 }
}

If Yes, I found a small trick in Intellij and DataGrip that could watch your SQL query results. It was always there but I overlooked it. Once I discovered it, it changed my life for debugging or development.

It’s not a trick actually 😅 but a feature built within the IntelliJ products. It could easily go ignored but let’s see how to use it

When you run any query you will see the Result window below:

You would see this clock icon in the toolbar that does all the trick. It was always there but I never thought it would be so useful 😍

When you click on it, the below options would be available. Each choice you select will refresh the query results after the selected delay.

A small tip to remember is to make sure you “Pin” 📌 the result window. When you do it, you could run other queries along with the watch window.

The below video shows a small demo of this feature. In the demo, I am watching the result of a SELECT query. When I update a record it is shown in the result window without running the SELECT query again.

5s seem long? What if I want to keep the watch period lower than 5s? You could use the Custom option provided. With this, you could also monitor your query changes at a 1s rate interval

Well, that's all for this post. I hope next time this small utility will come in handy for you. Don’t run the query every time to see the change. IntelliJ tools are a powerhouse of such small utilities which are very easily ignored by us.

Thanks! and Happy Coding! Don’t forget to follow me, like (❤️) for motivation and comment on some IntelliJ utilities that helped you! 🍻

This post is about why we need a separate transaction layer? How to extract it, and test it using Unit and Integration Test.

In my previous blog post, “A Billion Dollar Go Mistake,”. I discussed a common but naive mistake that developers make in Golang, which can lead to connection leaks. Although I offered several ways to fix this problem, one issue still bothered both myself and my readers. We can solve this problem by abstracting the transaction mechanism into a different layer. Many people reached out to me to suggest this solution.

So, my first curiosity question I want to reason:

"Do we really need a Transaction layer?"

I read several other blogs and GitHub codes. I noticed that many of them were great but did not include tests to prove that the abstraction layer works. But, we cannot confirm that the transaction layer works without tests. A connection leak may occur, and we cannot detect it by examining the code.

This led me to another important question:

"How can I prove that the new layer has no connection leaks?"

Testing is the only way to verify whether something works as expected. I decided to start with two straightforward options that came to my mind:

1. I will run some queries that change the table and verify that the code commits the data to the database.

2. I will use some mechanisms that expose the connection information. I could verify that the code closes the connections at the end.

The first approach is simple, and I have used it many times in my projects 😅. But, the second approach is completely unknown to me 👾. When I face difficulties in understanding how things work, I usually look at the source code of the language.

To resolve my problem, I looked for help in the standard database library code of Golang. I found something that could help me: the DBStats struct.

// DBStats contains database statistics.
type DBStats struct {
 MaxOpenConnections int // Maximum number of open connections to the database.

 // Pool Status
 OpenConnections int // The number of established connections both in use and idle.
 InUse           int // The number of connections currently in use.
 Idle            int // The number of idle connections.

 // Counters
 WaitCount         int64         // The total number of connections waited for.
 WaitDuration      time.Duration // The total time blocked waiting for a new connection.
 MaxIdleClosed     int64         // The total number of connections closed due to SetMaxIdleConns.
 MaxIdleTimeClosed int64         // The total number of connections closed due to SetConnMaxIdleTime.
 MaxLifetimeClosed int64         // The total number of connections closed due to SetConnMaxLifetime.
}

This is precisely the solution I was searching for. Upon completion of the transaction, the MaxOpenConnections and InUse counts should be 0. If this is not the case, it indicates a potential leak in the abstraction layer.

I am thrilled to have found the ideal solution to address the second scenario 🎉

To access the DBStats, we can use the Stats() method on the sql.DB instance, as shown below:

db, _ := sqlx.Open("postgres", "postgres://postgres:postgres@localhost:5432/postgres?sslmode=disable")
db.Stats()

The source code for the Stats() method is shown below. You could read more about it’s working and how go code records the information by following the source code here

// Stats returns database statistics.
func (db *DB) Stats() DBStats {
 wait := db.waitDuration.Load()

 db.mu.Lock()
 defer db.mu.Unlock()

 stats := DBStats{
  MaxOpenConnections: db.maxOpen,

  Idle:            len(db.freeConn),
  OpenConnections: db.numOpen,
  InUse:           db.numOpen - len(db.freeConn),

  WaitCount:         db.waitCount,
  WaitDuration:      time.Duration(wait),
  MaxIdleClosed:     db.maxIdleClosed,
  MaxIdleTimeClosed: db.maxIdleTimeClosed,
  MaxLifetimeClosed: db.maxLifetimeClosed,
 }
 return stats
}

All this is enough for me to implement the Transaction Layer. So, Let’s delve into the code now. But before we get started, let me introduce you to some of the libraries that I will be using in the code:

• Sqlx

• dockertest

• testify

• go-sqlmock

Interestingly, it’s hard to reproduce the connection leak scenario in a transaction layer. Trust me, I’ve tried and failed. 😂

I’ll show how to test the old code using DBStats assertions. With this example, people who will not abstract the transaction layer could update their tests to avoid any connection leaks. Later on, we’ll explore how to extract the transaction layer and test it.

package apptest

import (
 "context"
 "database/sql"
 "github.com/jmoiron/sqlx"
)

type Subscription struct {
 ID         int64        `db:"id"`
 Status     string       `db:"status"`
 CanceledAt sql.NullTime `db:"canceled_at"`
}

// ------------------------------ Repository ------------------------------

type srepo struct {
}

// GetSubscription fetches the subscription by id
func (r *srepo) GetSubscription(tx *sqlx.Tx, id int64) (Subscription, error) {
 var sub Subscription
 err := tx.Get(&sub, "SELECT * FROM subscription WHERE id = $1", id)
 if err != nil {
  return sub, err
 }

 return sub, nil
}

// CancelSubscription cancels a given subscription by setting canceled_at to now()
func (r *srepo) CancelSubscription(tx *sqlx.Tx, id int64) (Subscription, error) {
 var sub Subscription
 err := tx.Get(&sub, "UPDATE subscription SET canceled_at = NOW(), status='canceled' WHERE id = $1 RETURNING *", id)
 if err != nil {
  return sub, err
 }

 return sub, nil
}

// ------------------------------ Service ------------------------------

type Service struct {
 db   *sqlx.DB
 repo *srepo
}

func NewService(db *sqlx.DB, repo *srepo) *Service {
 return &Service{repo: repo, db: db}
}

func (s *Service) CancelSubscription(ctx context.Context, id int64) (*Subscription, error) {
 tx, err := s.db.BeginTxx(ctx, nil)
 if err != nil {
  return nil, err
 }

 defer func() {
  // !!! This would not work if the subscriptions is already canceled 
  // and the error is not returned
  if err != nil {
   _ = tx.Rollback()
   return
  }
 }()

 sub, err := s.repo.GetSubscription(tx, id)
 if err != nil {
  return nil, err
 }

 if sub.Status != "active" {
  return &sub, nil
 }

 if sub.CanceledAt.Valid {
  return &sub, nil
 }

 sub, err = s.repo.CancelSubscription(tx, id)
 if err != nil {
  return nil, err
 }

 err = tx.Commit()

 return &sub, err
}

What's wrong with the above code?

When the subscription is already cancelled it will return without error. When the function returns without error the connection neither rollback nor commits. This causes the connection leak.

Below is the integration test to see how we can catch the leak.

func Test_ConnectionLeak(t *testing.T) {
 pg, err := apptest.StartTestPostgres(t) // Please use the source code to learn more about this code
 require.NoError(t, err)

 _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS subscription (id serial PRIMARY KEY, status varchar(25) NOT NULL, canceled_at timestamp NULL)")
 require.NoError(t, err)

 _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('active', NULL)")
 require.NoError(t, err)

 _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('canceled', '2023-02-02 01:00:00')")
 require.NoError(t, err)

 subscription, err := NewService(pg.DB, &srepo{}).CancelSubscription(context.Background(), 2)
 require.NoError(t, err)

 stats := pg.DB.Stats()
 require.Equal(t, 0, stats.InUse, "expected no connections in use")
 require.Equal(t, 0, stats.MaxOpenConnections, "expected no max open connection")

 require.Equal(t, "canceled", subscription.Status)
 require.Equal(t, "2023-02-02 01:00:00 +0000 +0000", subscription.CanceledAt.Time.String())
}

I connect to a Postgres DB instance in docker and run the test. From the test result below you could see that the code has a transaction problem. The actual count InUse is 1. The connection is not closed at the end of the function call. This is what we will solve using the transaction abstraction layer in the next section.

=== RUN   Test_ConnectionLeak
    service-connection-leak_test.go:27: 
         Error Trace: /pkg/service-connection-leak_test.go:27
         Error:       Not equal: 
                      expected: 0
                      actual  : 1
         Test:        Test_ConnectionLeak
         Messages:    expected no connections in use
--- FAIL: Test_ConnectionLeak (6.69s)

Expected :0
Actual   :1
<Click to see difference>

FAIL

Extracting the Transaction Layer

To address the connection leak issue, one way could be to fix what is failing. Close the connection by manually testing the service and then reviewing the code. But that same issue could creep in again in future. The right way is to extract a transaction layer. The idea behind this extraction is simple: we provide a public method

func InTx(ctx context.Context, db *sqlx.DB, txFunc func(*TxWrap) error) (err error)

which accepts the transaction-based business logic in the txFunc parameter.

With this approach, the developer no longer has to manually handle transactions, as the InTx method abstracts the transaction mechanism away from the business logic. By passing the txFunc parameter to InTx, the developer can focus on the actual business operations, without worrying about the underlying transaction management.

I am keeping the code simple and avoiding any complications for blogging purposes.

package db

import (
 "context"
 "database/sql"
 "github.com/jmoiron/sqlx"
)

// TxWrap is a wrapper around sqlx.Tx that adds a context
// and redirects calls to methods like Get, Select to GetContext and SelectContext
// with the context it wraps.
type TxWrap struct {
 tx  *sqlx.Tx        // underlying transaction
 ctx context.Context // context to use for all calls
}

// Get is a wrapper around sqlx.Tx.GetContext
// that uses the context it wraps.
func (tx *TxWrap) Get(dest interface{}, query string, args ...interface{}) error {
 return tx.tx.GetContext(tx.ctx, dest, query, args...)
}

// Select is a wrapper around sqlx.Tx.SelectContext
// that uses the context it wraps.
func (tx *TxWrap) Select(dest interface{}, query string, args ...interface{}) error {
 return tx.tx.SelectContext(tx.ctx, dest, query, args...)
}

// IMPLEMENT OTHER RELATED sqlx Methods to use wrapped context

// InTx executes a function in a transaction.
// If the function returns an error, the transaction is rolled back.
// If the function panics, the transaction is rolled back and the panic is re-raised.
// If the function returns nil, the transaction is committed.
func InTx(ctx context.Context, db *sqlx.DB, txFunc func(*TxWrap) error) error {
 var err error
 ctx, cancel := context.WithCancel(ctx)
 defer cancel()

 tx, err := db.BeginTxx(ctx, nil)
 if err != nil {
  return err
 }

 txWrap := &TxWrap{
  tx:  tx,
  ctx: ctx,
 }

 defer func() {
  if p := recover(); p != nil {
   _ = txWrap.tx.Rollback()
   panic(p)
  }
  if err != nil {
   _ = txWrap.tx.Rollback()
   return
  }
  err = txWrap.tx.Commit()
 }()

 return txFunc(txWrap)
}

The InTx wraps around the actual sqlx.Tx in TxWrap struct and encapsulates the transaction management logic within the method. The InTx takes care of calling Begin, Commit and Rollback.

The TxWrap struct also contains a derived cancel context (ctx), which was discussed in my previous post to ensure the context cancellation happens at the end of the method call. Inside the InTx method, we have a defer block that handles three possible outcomes of the transaction logic:

1. Panic: In case of any unhandled exception, the transaction will automatically roll back.

2. Error: If any error occurs during the transaction, it will also be rolled back.

3. Success: If everything goes as expected, the transaction will be committed.

To provide further safety, we also implement the same Get and Select functions as in the sqlx library, but we proxy the calls to the context version of these functions. This ensures that any in-process requests are cancelled if the context is cancelled if a client cancels an HTTP request.

Now the real excitement begins !!! TESTS !!! 🚀 🥹

I am doing both approaches, the Unit test and the Integration Test. It is up to you what you want to use. My preference is the Integration test. They mimic behaviour close to real infrastructure.

Unit Test

For the unit test, I am usingsqlmock. I set up the expectation as per the 3 behaviours of our code and assert if the expectations are met. I also check the connection count resets 0 at the end. Unit Tests are pretty fast so we are also doing t.Parallel and initiating a new sqlmock for every test.

// ------------------------------ UNIT Test ------------------------------
func Test_Unit(t *testing.T) {
 t.Parallel()

 tests := []struct {
  name      string
  fn        func(tx *TxWrap) error
  setup     func(mock sqlmock.Sqlmock)
  wantErr   bool
  wantPanic bool
 }{
  {
   name: "success path",
   fn: func(tx *TxWrap) error {
    return nil
   },
   setup: func(mock sqlmock.Sqlmock) {
    mock.ExpectBegin()
    mock.ExpectCommit()
   },
  },
  {
   name: "failure path",
   fn: func(tx *TxWrap) error {
    return errors.New("some error")
   },
   setup: func(mock sqlmock.Sqlmock) {
    mock.ExpectBegin()
    mock.ExpectRollback()
   },
   wantErr: true,
  },
  {
   name: "panic",
   fn: func(tx *TxWrap) error {
    panic("some panic")
    return nil
   },
   setup: func(mock sqlmock.Sqlmock) {
    mock.ExpectBegin()
    mock.ExpectRollback()
   },
   wantPanic: true,
  },
 }
 for _, test := range tests {
  test := test
  t.Run(test.name, func(t *testing.T) {
   t.Parallel()

   db, mock, err := sqlmock.New()
   require.NoError(t, err)

   dbx := sqlx.NewDb(db, "sqlmock")

   if test.setup != nil {
      test.setup(mock)
   }

   // Only add this defer when we expect panic to take over the 
  // panic recovery and see if there is a valid error
   if test.wantPanic {
    defer func() {
     require.NotNil(t, recover())
    }()
   }

   err = InTx(context.Background(), dbx, test.fn)

   require.Equal(t, test.wantErr, err != nil)
   require.NoError(t, mock.ExpectationsWereMet())

   stats := dbx.Stats()
   require.Equal(t, 0, stats.InUse)
   require.Equal(t, 0, stats.MaxOpenConnections)
  })
 }
}

Integration Test

For the Integration test below, I am using Postgres DB. I create a dummy Employees table first. Then I execute some inserts and selects statements for different scenario. In the end, check if the layer closes the connection for every test.

A point to note here is that I am not doing t.Parallel. The shared connection between tests in the parallel run would be a problem. The InUse and MaxOpenConnections will never be 0. It is up to you how you want to do it. You can do a similar thing we did in the Unit test to create a separate connection for every test :

// ---------------------------------- INTEGRATION TEST -------------------------------------
type Employee struct {
 ID   int64  `db:"id"`
 Name string `db:"name"`
}

func Test_Integration(t *testing.T) {
 pg, err := apptest.StartTestPostgres(t)
 require.NoError(t, err)

 _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS employee (id serial PRIMARY KEY, name varchar(25) NOT NULL)")
 require.NoError(t, err)

 tests := []struct {
  name      string
  txfunc    func(tx *TxWrap) error
  wantErr   bool
  wantPanic bool
 }{
  {
   name: "success path",
   txfunc: func(tx *TxWrap) error {
    _, err := tx.Exec("INSERT INTO employee (name) VALUES ('John Doe')")
    return err
   },
  },
  {
   name: "failure path",
   txfunc: func(tx *TxWrap) error {
    var employee Employee
    err := tx.Get(&employee, "SELECT * FROM employee WHERE id = $1", 100)
    return err
   },
   wantErr: true,
  },
  {
   name: "panic",
   txfunc: func(tx *TxWrap) error {
    panic("some panic")

    return nil
   },
   wantPanic: true,
  },
 }
 for _, test := range tests {
  test := test
  t.Run(test.name, func(t *testing.T) {

   // Wrap the function in a defer to catch panics
   // and assert that the panic is not nil.
   defer func() {
    if test.wantPanic {
     require.NotNil(t, recover())
    }
    stats := pg.DB.Stats()
    require.Equal(t, 0, stats.InUse)
    require.Equal(t, 0, stats.MaxOpenConnections)
   }()

   err = InTx(context.Background(), pg.DB, test.txfunc)

   require.Equal(t, test.wantErr, err != nil)
  })
 }
}

Updating the Service

We have the new transaction layer ready. Let’s change our old code to use this new layer.

// ------------------------------ Repository ------------------------------

type txRepo struct {
}

// GetSubscription is a repository method that does not leak connections
// it uses *TxWrap to wrap the transaction
// it uses the context to cancel the transaction if the context is canceled
// but the context is inside the *TxWrap and not exposed to the service
func (r *txRepo) GetSubscription(tx *db.TxWrap, id int64) (Subscription, error) {
 var sub Subscription
 err := tx.Get(&sub, "SELECT * FROM subscription WHERE id = $1", id)
 if err != nil {
  return sub, err
 }

 return sub, nil
}

func (r *txRepo) CancelSubscription(tx *db.TxWrap, id int64) (Subscription, error) {
 var sub Subscription
 err := tx.Get(&sub, "UPDATE subscription SET canceled_at = NOW(), status='canceled' WHERE id = $1 RETURNING *", id)
 if err != nil {
  return sub, err
 }

 return sub, nil
}

// ------------------------------ Service ------------------------------

type txService struct {
 db   *sqlx.DB
 repo *txRepo
}

// CancelSubscriptionWithoutLeak is a service method that does not leak connections
// it uses InTx helper to wrap the transaction
func (s *txService) CancelSubscriptionWithoutLeak(ctx context.Context, id int64) (*Subscription, error) {
 var sub Subscription
 var err error

// So cool!!!!!!!! 🎸
 err = db.InTx(ctx, s.db, func(tx *db.TxWrap) error {
  sub, err = s.repo.GetSubscription(tx, id)
  if err != nil {
   return err
  }

  if sub.Status != "active" {
   return nil
  }

  if sub.CanceledAt.Valid {
   return nil
  }

  sub, err = s.repo.CancelSubscription(tx, id)
  if err != nil {
   return err
  }

  return nil
 })

 return &sub, err
}

func Test_NoConnectionLeak(t *testing.T) {
 pg, err := apptest.StartTestPostgres(t)
 require.NoError(t, err)

 _, err = pg.DB.Exec("CREATE TABLE IF NOT EXISTS subscription (id serial PRIMARY KEY, status varchar(25) NOT NULL, canceled_at timestamp NULL)")
 require.NoError(t, err)

 _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('active', NULL)")
 require.NoError(t, err)

 _, err = pg.DB.Exec("INSERT INTO subscription (status, canceled_at) VALUES ('canceled', '2023-02-02 01:00:00')")
 require.NoError(t, err)

 subscription, err := NewTxService(pg.DB, &txRepo{}).CancelSubscriptionWithoutLeak(context.Background(), 2)
 require.NoError(t, err)

 stats := pg.DB.Stats()
 require.Equal(t, 0, stats.InUse, "expected no connections in use")
 require.Equal(t, 0, stats.MaxOpenConnections, "expected no max open connection")

 require.Equal(t, "canceled", subscription.Status)
 require.Equal(t, "2023-02-02 01:00:00 +0000 +0000", subscription.CanceledAt.Time.String())
}

-------------

=== RUN   Test_NoConnectionLeak
--- PASS: Test_NoConnectionLeak (5.61s)
PASS

The test above is the exact test we used in our connection leak example. I have to tweak it for the new service and new imports. Everything else in the test is the same.

You could see when we changed the service to use the new transaction layer our same test is green. 🥳🥳🥳

Do We Need A Transaction Layer?

It was tough for me to create a connection leak with the transaction layer in the old code. It is close to impossible that anything could go wrong.

Although, I felt that testing would be a challenge. The DBStruct provided an easy way for testing.

With different business operations within a transaction, a single line could cause problems and issues in production. But a separate battle-tested layer would safeguard against any issues. Plus, the same layer could be shared with different flows to avoid code repetition.

So, if you haven’t yet extracted that transaction logic please do! You could write your library or use something off the shelf. I am also going to publish the lib on my GitHub which I would share with you 🚀

Conclusion

With this proof, I am very confident that I will not cause another transaction-related production issue. And I hope this blog post helps you to learn something new about Golang.

Some of my initial assumptions were proven wrong. The tests could allow me to answer that the layer works and there will be no connection leak.

I want to thank all my readers 🙏. I feel a lot motivated by your replies and suggestions.

It feels good when people are reading the blog and in some way, I am helping the community back. If you have any suggestions or want to reach out to me please use the AMA

I am exploring more production-level stuff in Golang, and things that I am working on. If you like to be part of this journey please follow me on Medium or Hashnode and subscribe to my posts. You could also reach out to me on other platforms:

• Twitter

• Github

• LinkedIn

This post is about a real problem I faced in my project. I merged two blog post into one so this is a bit longer post, please read till the end. Thank you!

I hope after reading this post you will be able to avoid the same mistake we did in our project 😅. This small mistake didn’t cost us a billion dollars. But, It did cost us a few thousand, and the Prometheus alerts saved us.

But, it could cost you a Billion if you are not careful 💸. I learn my lessons through mistakes, this is something that I will never forget in future.

The scenario is a simple database transaction. We all have used database transactions at least once in our programming life. If you don’t know how transactions work you could read the docs here for Postgres.

Our service handles the management of some Subscriptions in the database. All the changes to the rows happen within a transaction. The service is in Golang and the flow is like this::

• Start the transaction

• Fetch the record by ID

• Confirm if the operation for the change is possible

• If Yes, then update the record

• Commit the transaction

• For any error, revert the transaction

To fetch a record and acquire a lock on the record so that other flows would wait to update, we use SELECT ... FOR UPDATE query to fetch the record from Postgres and lock it.

Note: We use sqlx lib for database access.

The repository code is like this:

func (r *fetcher) GetSubscription(tx *sqlx.Tx, id uuid.UUID) (*model.Subscription, error) {
    var subscription model.Subscription
    err := tx.Get(&subscription, `
        SELECT * FROM subscriptions
        WHERE id = $1
        FOR UPDATE
    `, id)
    if err != nil {
        return nil, err
    }

    return &subscription, nil
}

The service code is like this:

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, error) {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }

    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()

    subscription, err := s.fetcher.GetSubscription(tx, id)
    if err != nil {
        return nil, err
    }

    if subscription.CancelledAt != nil {
        return subscription, nil
    }

    subscription.CancelledAt = time.Now()

    err = s.updater.UpdateSubscription(tx, subscription)
    if err != nil {
        return nil, err
    }

    err = tx.Commit()
    if err != nil {
        return nil, err
    }

    return subscription, nil
}

Problem

All client requests were timing out, and the DB connection spiked over 1.2k connections 😅

Not a single request was able to complete the operation. It was a stop-the-world event 🤣

Why?

The issue happens when we try to cancel a subscription that is already cancelled. If the subscription is already cancelled we return the subscription without doing any changes, but we are not releasing the lock on the record.

The reason is defer function calls tx.Rollback() only when there is an error. This would cause the lock to be active until the transaction commits or roll back. But since we are not doing any of the two things, the lock is held until the transaction times out.

If the service is having high traffic, the transactions could not timeout. This will cause the code to create new connections to the database and exhaust the connection pool.

Fix

1. Release the lock in every if condition.

// Error handling is omitted for brevity
if subscription.CancelledAt != nil {
    _ = tx.Rollback() // release the lock

    return subscription, nil
}

This is the simplest way to fix the issue. But this would need you to roll back in every if condition. And if you forget to roll back in any of the if conditions, you will have the same issue.

2. Rollback the transaction in the defer function for every case.

// Error handling is omitted for brevity
defer func() {
   _ = tx.Rollback()
}()

This would release the lock in every case. For committed transactions and as per the tracing, we see some milliseconds the code spends to roll back the transactions. In the case of committed transactions, the rollback does nothing. But this is a better way to fix the issue.

3. Commit the transaction in the defer function for every case.

defer func() {
  if err != nil {
    _ = tx.Rollback()
  }

  commitErr := tx.Commit()
  // handle commitErr
}

If there are no changes the transaction will commit without any change. If there is any error only then the rollback would happen and will release the lock.

But, this affects the readability of the code. Your commit is happening in the defer function, and that's an extra pointer you have to keep in mind while reading the code.

Service Layer — Return Error 👾

The problem with the service is that for validations, it is not returning any error. The other way to fix this issue is to return an error from the if condition. This would make the Rollback happen in defer function.

This is a very clean way. A service should have only one responsibility. Either it completes the operation or it returns an error. It should return an error in case of validation failure. This is a good practice and something we missed in our project. We fixed the issue later by returning an error from the if condition.

This change also helps the handler to decide what HTTP status code to return. Which is really helpful as we can return 400 Bad Request for validation errors.

This is how the code would look after refactoring

var ErrSubscriptionAlreadyCancelled = errors.New("subscription already cancelled")

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, error) {
    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }

    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()

    subscription, err := s.fetcher.GetSubscription(tx, id)
    if err != nil {
        return nil, err
    }
    if subscription.CancelledAt != nil {
        return subscription, ErrSubscriptionAlreadyCancelled
    }

    subscription.CancelledAt = time.Now()
    err = s.updater.UpdateSubscription(tx, subscription)
    if err != nil {
        return nil, err
    }
    err = tx.Commit()
    if err != nil {
        return nil, err
    }
    return subscription, nil
}

My Take

My personal preference is the second option. But you could choose any of the three options based on your preference. I am choosing the second option because I am sure that whatever happens in the flow at the end, the transaction revert will happen. Yes, it would cost me a few milliseconds in case of a committed transaction, but compared to the loss to the business, it’s worth it.

Keep in mind that the service layer has only one responsibility. Either it completes the operation or it returns an error.

You may be thinking that a better approach is to have an abstraction that takes care of transactions. I agree, and would also have a separate post on that. But, I also think that we can avoid the complexity of abstraction if the code is stable and not changes too often.

The golang official post for transactions also supports the reasoning. Check the code snippet here. The office post also uses the second option.

// code snippet from the golang official post
func CreateOrder(ctx context.Context, albumID, quantity, custID int) (orderID int64, err error) {
// Create a helper function for preparing failure results.
  fail := func (err error) (int64, error) {
    return fmt.Errorf("CreateOrder: %v", err)
  }
  // Get a Tx for making transaction requests.
  tx, err := db.BeginTx(ctx, nil)
  if err != nil {
    return fail(err)
  }
  // Defer a rollback in case anything fails.
  defer tx.Rollback()

  ... // other code is omitted for brevity

  // Commit the transaction.
  if err = tx.Commit(); err != nil {
    return fail(err)
  }

  // Return the order ID.
  return orderID, nil
}

There are two more ways to fix the problem. This will also provide one extra layer of safety. In my opinion, having multiple safety gates is always better.

If one fails there is another one to safeguard with any issues. The business-critical services with an extra layer will protect against any monetary loss.

Excited to know more?

I do this kind of dance when I find multiple ways to solve any problem :D

Using a context

We could use a cancel context to release the lock at the end of the transaction. Context is very helpful in many use cases.

Let’s see how to do things with context

func (s *service) CancelSubscription(ctx context.Context, id uuid.UUID) (*model.Subscription, err error) {
    ctx, cancel := context.WithCancel(ctx)
    defer cancel()

    tx, err := s.db.BeginTxx(ctx, nil)
    if err != nil {
        return nil, err
    }

    defer func() {
        if err != nil {
            tx.Rollback()
        }
    }()

    subscription, err := s.fetcher.GetSubscription(ctx, tx, id)
    if err != nil {
        return nil, err
    }

    if subscription.CancelledAt != nil {
        return subscription, nil
    }

    subscription.CancelledAt = time.Now()

    err = s.updater.UpdateSubscription(ctx, tx, subscription)
    if err != nil {
        return nil, err
    }

    err = tx.Commit()
    if err != nil {
        return nil, err
    }

    return subscription, nil
}

In the above code, we create a new cancel context from the parent context. If the DB transaction is not released either using Rollback or Commit. A defer cancel() happens at the end when the call returns.

The cancel call will notify the transaction that the operation is complete. The go runtime will close the transaction.

The defer cancel() call is important. If you miss that then the problem will still persist

At the go source code level transaction begin call starts a background go routine. The go routine monitors context Done signal. Let's see how it happens using context as shown here

func (db *DB) beginDC(ctx context.Context, dc *driverConn, release func(error), opts *TxOptions) (tx *Tx, err error) {
     var txi driver.Tx
     keepConnOnRollback := false
     withLock(dc, func() {
      _, hasSessionResetter := dc.ci.(driver.SessionResetter)
      _, hasConnectionValidator := dc.ci.(driver.Validator)
      keepConnOnRollback = hasSessionResetter && hasConnectionValidator
      txi, err = ctxDriverBegin(ctx, opts, dc.ci)
     })
     if err != nil {
      release(err)
      return nil, err
     }

     ctx, cancel := context.WithCancel(ctx)
     tx = &Tx{
      db:                 db,
      dc:                 dc,
      releaseConn:        release,
      txi:                txi,
      cancel:             cancel,
      keepConnOnRollback: keepConnOnRollback,
      ctx:                ctx,
     }
     go tx.awaitDone() // GO WAITS FOR THE CONTEXT TO DONE
     return tx, nil
}

And inside the tx.awaitDone

func (tx *Tx) awaitDone() {
 // Wait for either the transaction to be committed or rolled
 // back, or for the associated context to be closed.
 <-tx.ctx.Done()

 // Discard and close the connection used to ensure the
 // transaction is closed and the resources are released.  This
 // rollback does nothing if the transaction has already been
 // committed or rolled back.
 // Do not discard the connection if the connection knows
 // how to reset the session.
 discardConnection := !tx.keepConnOnRollback
 tx.rollback(discardConnection)  // AT THE END THE TX WILL BE ROLLBACKED
}

So transactions with a cancel context make more sense. Do not use the context coming from the HTTP request. HTTP context works like a client-side timeout. The service layer has no control over cancelling the HTTP context.

A timeout context could also work but that will not be that reliable. Finding the right timeout value is a difficult task in itself.

Server-side timeout

You set an optimal timeout for the request to complete and if the request is not completed within that time, you cancel the operation and return an error.

server := &http.Server{
    Addr:         ":8080",
    Handler:      router,
    WriteTimeout: 2 * time.Second,
}

In the above code, we are setting a server-side timeout of 2 seconds. If the entire request processing is not completed within 2 seconds, we cancel the operation and return an error to the client. With this approach, you don’t need to create a timeout context for each request. But, you would still need to fine-tune the timeout value based on the traces and load testing.

Conclusion

I hope you found this post useful. For customer-facing services, having a tiny bug could cause a big monetary impact. Securing code with enough safety nets is something we should do. And, I hope through this post I made a lot of people aware of what worse could happen if the transactions are leaking and how easy it is to fix them. 😊

To conclude my take I recommend using rollback of the transaction in defer always. Finding a reasonable timeout for context and server-side timeout is difficult. So, it doesn't matter if there is an error or not defer with rollback.

Cancel context is the best approach for most application types. And would serve most of the cases. The context in general has a lot of use cases and I would recommend you read more about it.

Also, for an extra layer of safety, I would recommend keeping a Server side timeout for HTTP services.

You would be also thinking about the client-side timeout. Those are not reliable enough. For example, a client could keep the timeout for a long duration and with a high load, the server could keep the connection open for a long time. This could again lead to the same problem we are trying to solve.

References:

• Go Concurrency Patterns: Context

• Contexts and structs

I hope my team does a similar kind of dance one day 😅

Edits:

• Fixed the context code with using cancel context instead of timeout context

• Added the source code of golang database layer on how the cancellation is managed

I am working with Change Data Capture using Debezium recently and this docker-compose helps me to write the consumer application that will process the changes happening at the DB level.

In this post, I will be setting up the Debezium connector for Postgres with all the changes required to allow Debezium to capture the changes. In addition, I will also provide the details on how to make this configuration on the AWS RDS if you are going to use the Debezium Connector for your next project.

The following services would be needed to make the Debezium and Postgres work locally:

• Kafka Broker - localhost:9092

• Zookeeper - localhost:2181

• Postgres - localhost:5432

• Debezium Connector - localhost:8083

• Schema Registry - localhost:8081

• Debezium UI - localhost:8080

• Rest-Proxy - This is optional, but helps with checking cluster metadata, topics etc - localhost:8082

Starting Docker Compose

Below is the docker-compose file I use to start the stack:

version: "3.9"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:7.3.1
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    healthcheck:
      test: echo srvr | nc zookeeper 2181 || exit 1
      start_period: 10s
      retries: 20
      interval: 10s
  broker:
    image: confluentinc/cp-kafka:7.3.1
    hostname: broker
    container_name: broker
    depends_on:
      zookeeper:
        condition: service_healthy
    ports:
      - "29092:29092"
      - "9092:9092"
      - "9101:9101"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://broker:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "true"
      KAFKA_JMX_PORT: 9101
      KAFKA_JMX_HOSTNAME: localhost
    healthcheck:
      test: nc -z localhost 9092 || exit -1
      start_period: 15s
      interval: 5s
      timeout: 10s
      retries: 10
  debezium:
    image: debezium/connect:latest
    restart: always
    container_name: debezium
    hostname: debezium
    depends_on:
      postgres:
        condition: service_healthy
      broker:
        condition: service_healthy
    ports:
      - "8083:8083"
    environment:
      BOOTSTRAP_SERVERS: broker:29092
      GROUP_ID: 1
      CONFIG_STORAGE_TOPIC: connect_configs
      STATUS_STORAGE_TOPIC: connect_statuses
      OFFSET_STORAGE_TOPIC: connect_offsets
      KEY_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      VALUE_CONVERTER: org.apache.kafka.connect.json.JsonConverter
      ENABLE_DEBEZIUM_SCRIPTING: "true"
    healthcheck:
      test: ["CMD", "curl", "--silent", "--fail", "-X", "GET", "http://localhost:8083/connectors"]
      start_period: 10s
      interval: 10s
      timeout: 5s
      retries: 5

  schema-registry:
    image: confluentinc/cp-schema-registry:7.3.1
    hostname: schema-registry
    container_name: schema-registry
    depends_on:
      broker:
        condition: service_healthy
    ports:
      - "8081:8081"
    environment:
      SCHEMA_REGISTRY_HOST_NAME: schema-registry
      SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: broker:29092
      SCHEMA_REGISTRY_LISTENERS: http://0.0.0.0:8081

    healthcheck:
      start_period: 10s
      interval: 10s
      retries: 20
      test: curl --user superUser:superUser --fail --silent --insecure http://localhost:8081/subjects --output /dev/null || exit 1

  rest-proxy:
    image: confluentinc/cp-kafka-rest:7.3.1
    depends_on:
      broker:
        condition: service_healthy
    ports:
      - "8082:8082"
    hostname: rest-proxy
    container_name: rest-proxy
    environment:
      KAFKA_REST_HOST_NAME: rest-proxy
      KAFKA_REST_BOOTSTRAP_SERVERS: 'broker:29092'
      KAFKA_REST_LISTENERS: "http://0.0.0.0:8082"

  debezium-ui:
    image: debezium/debezium-ui:latest
    restart: always
    container_name: debezium-ui
    hostname: debezium-ui
    depends_on:
      debezium:
        condition: service_healthy
    ports:
      - "8080:8080"
    environment:
      KAFKA_CONNECT_URIS: http://debezium:8083


  postgres:
      image: postgres:latest
      restart: always
      container_name: postgres
      hostname: postgres
      ports:
        - "5432:5432"
      environment:
        POSTGRES_USER: postgres
        POSTGRES_PASSWORD: postgres
        POSTGRES_DB: movies_db
      command: ["postgres", "-c", "wal_level=logical"]
      healthcheck:
        test: ["CMD", "psql", "-U", "postgres", "-c", "SELECT 1"]
        interval: 10s
        timeout: 5s
        retries: 5
      volumes:
        - ./scripts:/docker-entrypoint-initdb.d

For Debezium to work with Postgres, Postgres needs to have the logical replication enabled and if you observe the line command: ["postgres", "-c", "wal_level=logical"] we are configuring the Postgres DB to start with wal_level as logical.

If we don't do this step, then debezium would not be able to capture the changes happening on Postgres. The default wal_level is replica.

Now we have our docker-compose file ready let's start all the service

 docker compose -f docker-compose.yml up -d
[+] Running 8/8
 ⠿ Network postgres_default   Created                                                                                                                                                                          0.1s
 ⠿ Container postgres         Healthy                                                                                                                                                                         12.8s
 ⠿ Container zookeeper        Healthy                                                                                                                                                                         11.8s
 ⠿ Container broker           Healthy                                                                                                                                                                         22.6s
 ⠿ Container debezium         Healthy                                                                                                                                                                         44.2s
 ⠿ Container rest-proxy       Started                                                                                                                                                                         23.1s
 ⠿ Container schema-registry  Started                                                                                                                                                                         23.1s
 ⠿ Container debezium-ui      Started                                                                                                                                                                         44.6s

As we could see all the containers have started without any errors. If you are running this command for the first time it would take some time to download all the docker images but the later executions will be faster

Configure Postgres Connector

At this point, our Debezium connector is running but it doesn't have any task to read changes happening on the Postgres DB. We need to register the Postgres connector using HTTP API so that debezium could read the transaction logs from the server.

It's easy to register the connector, there is a sample connector config already present in the repo. Execute the below curl and we would see the connector registered

curl -X POST --location "http://localhost:8083/connectors" -H "Content-Type: application/json" -H "Accept: application/json" -d @connector.json

{
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.dbname": "movies_db",
        "database.history.kafka.bootstrap.servers": "kafka:9092",
        "database.history.kafka.topic": "schema-changes.movies",
        "database.hostname": "postgres",
        "database.password": "postgres",
        "database.port": "5432",
        "database.server.name": "postgres",
        "database.user": "postgres",
        "name": "movies-db-connector",
        "plugin.name": "pgoutput",
        "table.include.list": "public.movies",
        "tasks.max": "1",
        "topic.creation.default.cleanup.policy": "delete",
        "topic.creation.default.partitions": "1",
        "topic.creation.default.replication.factor": "1",
        "topic.creation.default.retention.ms": "604800000",
        "topic.creation.enable": "true",
        "topic.prefix": "postgres"
    },
    "name": "movies-db-connector",
    "tasks": [],
    "type": "source"
}

List Kafka Topics

If there was no issue running the above steps we could confirm that our connector is working fine by checking if the topic is created for movies table by the connector.

❯ kafka-topics --bootstrap-server localhost:9092 --list
__consumer_offsets
_schemas
connect_configs
connect_offsets
connect_statuses
postgres.public.movies

As you could see from the above command we have postgres.public.movies a topic created by debezium. The topic name pattern is of a type topic_prefix_config.schema.table so the prefix is what we configured and the other part is fetched by the debezium from the database schema

Reading the data

Now we know that the topic is created next we would check the data available on the topic.

There would be data present in the topic because when the connector starts it takes an initial snapshot of the database table. This is a default config named snapshot.mode which we didn't configure but is set to initial which means that the connector will do a snapshot on the initial run when it doesn't find the last known offset from the transaction log available for the database server, to understand more about this configuration and others read more here.

kafka-console-consumer --bootstrap-server localhost:9092 --topic postgres.public.movies --from-beginning
{
    "schema": {
        "type": "struct",
        "fields": [
            {
                "type": "struct",
                "fields": [
                    {
                        "type": "int64",
                        "optional": false,
                        "field": "id"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "title"
                    },
                    {
                        "type": "int32",
                        "optional": false,
                        "field": "year"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "director"
                    },
                    {
                        "type": "struct",
                        "fields": [
                            {
                                "type": "int32",
                                "optional": false,
                                "field": "scale"
                            },
                            {
                                "type": "bytes",
                                "optional": false,
                                "field": "value"
                            }
                        ],
                        "optional": false,
                        "name": "io.debezium.data.VariableScaleDecimal",
                        "version": 1,
                        "doc": "Variable scaled decimal",
                        "field": "rating"
                    }
                ],
                "optional": true,
                "name": "postgres.public.movies.Value",
                "field": "before"
            },
            {
                "type": "struct",
                "fields": [
                    {
                        "type": "int64",
                        "optional": false,
                        "field": "id"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "title"
                    },
                    {
                        "type": "int32",
                        "optional": false,
                        "field": "year"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "director"
                    },
                    {
                        "type": "struct",
                        "fields": [
                            {
                                "type": "int32",
                                "optional": false,
                                "field": "scale"
                            },
                            {
                                "type": "bytes",
                                "optional": false,
                                "field": "value"
                            }
                        ],
                        "optional": false,
                        "name": "io.debezium.data.VariableScaleDecimal",
                        "version": 1,
                        "doc": "Variable scaled decimal",
                        "field": "rating"
                    }
                ],
                "optional": true,
                "name": "postgres.public.movies.Value",
                "field": "after"
            },
            {
                "type": "struct",
                "fields": [
                    {
                        "type": "string",
                        "optional": false,
                        "field": "version"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "connector"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "name"
                    },
                    {
                        "type": "int64",
                        "optional": false,
                        "field": "ts_ms"
                    },
                    {
                        "type": "string",
                        "optional": true,
                        "name": "io.debezium.data.Enum",
                        "version": 1,
                        "parameters": {
                            "allowed": "true,last,false,incremental"
                        },
                        "default": "false",
                        "field": "snapshot"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "db"
                    },
                    {
                        "type": "string",
                        "optional": true,
                        "field": "sequence"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "schema"
                    },
                    {
                        "type": "string",
                        "optional": false,
                        "field": "table"
                    },
                    {
                        "type": "int64",
                        "optional": true,
                        "field": "txId"
                    },
                    {
                        "type": "int64",
                        "optional": true,
                        "field": "lsn"
                    },
                    {
                        "type": "int64",
                        "optional": true,
                        "field": "xmin"
                    }
                ],
                "optional": false,
                "name": "io.debezium.connector.postgresql.Source",
                "field": "source"
            },
            {
                "type": "string",
                "optional": false,
                "field": "op"
            },
            {
                "type": "int64",
                "optional": true,
                "field": "ts_ms"
            },
            {
                "type": "struct",
                "fields": [
                    {
                        "type": "string",
                        "optional": false,
                        "field": "id"
                    },
                    {
                        "type": "int64",
                        "optional": false,
                        "field": "total_order"
                    },
                    {
                        "type": "int64",
                        "optional": false,
                        "field": "data_collection_order"
                    }
                ],
                "optional": true,
                "name": "event.block",
                "version": 1,
                "field": "transaction"
            }
        ],
        "optional": false,
        "name": "postgres.public.movies.Envelope",
        "version": 1
    },
    "payload": {
        "before": null,
        "after": {
            "id": 1,
            "title": "The Shawshank Redemption",
            "year": 1994,
            "director": "Frank Darabont",
            "rating": {
                "scale": 1,
                "value": "XQ=="
            }
        },
        "source": {
            "version": "2.1.1.Final",
            "connector": "postgresql",
            "name": "postgres",
            "ts_ms": 1675103983563,
            "snapshot": "first",
            "db": "movies_db",
            "sequence": "[null,\"26612016\"]",
            "schema": "public",
            "table": "movies",
            "txId": 744,
            "lsn": 26612016,
            "xmin": null
        },
        "op": "r",
        "ts_ms": 1675103983739,
        "transaction": null
    }
}
----- more data ommitted for readability purpose -----

Cleanup

By running the below command all the services will be shut down and the resources would be released

docker compose -f docker-compose.yml -v down

Here we specify -v option to also remove any volumes created by the docker-compose, this will also free the disk space allocated to the containers

Debezium UI

Debezium also provides a control dashboard. You could use it to add a new connector instead of using the REST API. The dashboard also provides the functionality to Pause or Restart the debezium task and also to delete the connector

Go to http://localhost:8080 and you will be able to access the dashboard as shown below

The Dashboard also allows you to add new connectors for other database providers

AWS RDS

Setting up docker-compose is easy because everything is local and we could use the configuration described by the Postgres documentation.

On AWS, it's a different story, on AWS we have to configure rds.logical_replication property in the parameter groups of your Postgres instance. The default value for this property is 0 which means the logical replication is disabled by default, however, setting it to 1 will enable the logical replication. An important thing that you should know is that when you change this property on AWS you would need to restart your AWS RDS Postgres instance. If your application is in production that means you would need some downtime to enable the replication. You could read more about this and the steps to do it in AWS RDS here

This brings us to the end of this small and quick setup of Postgres and Debezium using Docker Compose. Hope, next time you are building any CDC-related consumer this docker-compose setup will help you to run the infrastructure locally without any hassle.

If you like the blog, please don't forget to like the post and subscribe to the Newsletter so that you would get the new blog posts and updates directly to your mailbox.

Code

• Github

I was working on sending some telemetry data from my Golang web application to Prometheus and then creating a Grafana dashboard out of it. But to test the flow I needed a local setup of Prometheus + Grafana so that I can check if the metrics are right and that I am building the right PromQL query to create the dashboard.

I am maintaining a Github repo for all the docker-compose setups I require for my local testing and now Prometheus + Grafana is a new addition to it. If you also need a similar setup and save your setup time in future and focus more on building things refer to this post or my Github Repository.

Clone the repo: https://github.com/ninadingole/docker-compose-stacks

Then go to prometheus-grafana folder and run docker-compose up -d.

This will start Prometheus on http://localhost:9090 and Grafana on http://localhost:3000.

There is also a prometheus.yml the configuration file which you can use to add the local apps that you want to scrape,

Note: if your application is running inside a docker then use host.docker.internal as your hostname with the port to scrape the target.

global:
  scrape_interval:     15s
  evaluation_interval: 15s

rule_files:
  # - "first.rules"
  # - "second.rules"

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']
  - job_name: app
    scrape_interval: 5s
    static_configs:
      - targets: ['host.docker.internal:10088']

Once you have added the configs to connect Grafana to Prometheus like the above image, you are ready with an end-to-end setup on your local.

If you also have your own docker-compose stack setups that you use during your development please don't hesitate to share with me by sending me a PR to the repository.

I hope you will find this docker-compose configuration pretty useful and saves you time. Please subscribe to the newsletter to get more articles delivered right to your inbox.

Thanks and Happy Coding!

Reference

• github

Recently, I came across the same situation where I was looking for a way to change the response of the service that I was consuming and instead of mocking the entire service using a tool like Postman or wire mock, I wanted to hit the actual service and then change a few fields to see if my error handling works as per the expectations.

Normally, we do write an integration test where we mock the service response using wiremock and then write the test for such edge cases. However, sometimes when we do manual testing and we want a way to mock the service response we need some mechanism to intercept the service response and modify the response. This is where Burp suites will help, Burp suites acts like a proxy that will call the actual service on behalf of you and then provides you with tools that can help to easily modify the response and send it back to your application.

For this post, I will be consuming themoviesdb.org API's to see details of "Fight Club" movie, and then I will write an application in golang which will connect to Burp Suite, there we modify the response using Burpsuite UI and print the modified response in the console which is received by our application.

package main
import (
    "crypto/tls"
    "encoding/json"
    "fmt"
    "net/http"
    "net/url"
    "os"
)
func main() {
    proxyURL, e := url.Parse(os.Getenv("PROXY"))
    if e != nil {
        panic(e)
    }
    client := &http.Client{
        //Timeout: 20 * time.Second,
        Transport: &http.Transport{
            Proxy: http.ProxyURL(proxyURL),
            TLSClientConfig: &tls.Config{InsecureSkipVerify:true},
        },
    }
    get, err := client.Get(fmt.Sprintf("https://api.themoviedb.org/3/movie/550?api_key=%s", os.Getenv("KEY")))
    if err != nil {
        panic(err)
    }
    var data map[string]interface{}
    err = json.NewDecoder(get.Body).Decode(&data)
    if err != nil {
        panic(err)
    }
    fmt.Printf("Response: %+v",data)
}

As you can see in the above code, we create an HTTP client with custom Transport set to proxy URL which is configured in the environment setting of the application, for burp suite the URL is http://localhost:8080. The TLSClient config is configured to skip ssl certificate verification using InsecureSkipVerify:true. Then we do a normal GET request to the api which will get the details of the movie id 550.

Let's now start the burp suite and configure the intercept rule to only run when the domain matches themoviedb.org:

Then start the interceptor:

Now, run the go application and you will see that the request is captured by the burp suite as follows:

After you see the above screen click on the Action button and click Do intercept with Response to this request:

After clicking the Forward button, burp suite will call the actual service and will open the editor where you can edit the response, this movie is "Flight Club" movie and I have changed the movie name to Fight Pub and rating from 8.4 to 9.4. Then click the forward button again to forward the response to the golang app.

Once the application receives this response we just print it to the console and you could see that the name of the movie is now Fight Pub and the rating is 9.4 which is what we modified in the response.

Response: map[adult:false 
backdrop_path:/mMZRKb3NVo5ZeSPEIaNW9buLWQ0.jpg belongs_to_collection:<nil> 
budget:6.3e+07 
genres:[map[id:18 name:Drama]] 
homepage:http://www.foxmovies.com/movies/fight-club id:550 imdb_id:tt0137523 
original_language:en 
original_title:Fight Pub 
overview:A ticking-time-bomb insomniac and a slippery soap salesman channel primal male aggression into a shocking new form of therapy. Their concept catches on, with underground "fight clubs" forming in every town, until an eccentric gets in the way and ignites an out-of-control spiral toward oblivion. 
popularity:58.844 
poster_path:/adw6Lq9FiC9zjYEpOqfq03ituwp.jpg 
production_companies:[map[id:508 logo_path:/7PzJdsLGlR7oW4J0J5Xcd0pHGRg.png name:Regency Enterprises origin_country:US] map[id:711 logo_path:/tEiIH5QesdheJmDAqQwvtN60727.png name:Fox 2000 Pictures origin_country:US] map[id:20555 logo_path:/hD8yEGUBlHOcfHYbujp71vD8gZp.png name:Taurus Film origin_country:DE] map[id:54051 logo_path:<nil> name:Atman Entertainment origin_country:] map[id:54052 logo_path:<nil> name:Knickerbocker Films origin_country:US] map[id:25 logo_path:/qZCc1lty5FzX30aOCVRBLzaVmcp.png name:20th Century Fox origin_country:US] map[id:4700 logo_path:/A32wmjrs9Psf4zw0uaixF0GXfxq.png name:The Linson Company origin_country:]] 
production_countries:[map[iso_3166_1:DE name:Germany] map[iso_3166_1:US name:United States of America]] 
release_date:1999-10-15 
revenue:1.00853753e+08 
runtime:139 
spoken_languages:[map[iso_639_1:en name:English]] 
status:Released 
tagline:Mischief. Mayhem. Soap. 
title:Fight Pub 
video:false 
vote_average:9.4 
vote_count:17753]

So, thats how you can connect your application to the burp suite and can intercept the response of the services to manually test the application for different cases.

I hope this post helps you to understand how to configure an HTTP client in your code to connect to Burp suite, I have used golang for this post because I am currently working on it. However, if you are working on any other programming language the HTTP Api's will have some way to configure the proxy as like I have done in my golang code.

Happy Coding!

In short and crisp, tests gets divided into three categories: unit test, integration/service test, and e2e/UI tests. The unit test validates the functionality of each class and written on per class basis, integration test checks for the functionality of module with external systems like database, web service, and UI/e2e test the entire business flow or UI flow.

As you can see from the test pyramid, we should have more integration test than e2e test and more unit test than the integration test. The reason for having less integration and e2e tests is "time"; Integration test and e2e test interacts typically with other systems which are external or internal like in-memory DB or clusters. These tests take time to execute and are often slower.

And not to mention we have our CD/CI pipelines which run these tests and generate our code artifact. We also want our CI to provide us faster feedback by running these test individually, so we don't have to wait longer if there is an issue at the unit level.

Currently, I am working on a Big Data project which includes frameworks like Spark, HBase, Kafka, HDFS, etc. Some of our unit tests are independent of any spark code, and some unit test does require spark context, but they do not interact with external systems like HDFS and HBase. Moreover, integration test, runs an in-memory HBase cluster, in-memory mini dfs cluster and spark of course :P. Now, these integration test shares the single running clusters amongst them and executes sequentially.

We have SBT as the build tool and, we have disabled the parallel execution of tests which makes the entire test suite including the unit test to run slower. I am struggling with that sometimes have to wait for 10-15mins to get the feedback about my code changes, running unit tests also runs integration test which takes time for cluster initialization of HBase and HDFS, and I was seriously looking for some alternative to this issue. Seldom I do comment out my integration suite code only to run the unit test which is again a wrong way. We need some mechanism to organize our tests such that we can execute them separately as a suite of unit, integration and e2e tests and, we only have parallel execution disabled for integration and e2e tests. It would provide developers with quick feedback by just running the unit tests.

In this post, I am going to share my findings on how to organize the tests and configure sbt (v1.2.8) such that it enables us to run these different scala test independently without any code hacks or commenting your test cases which is a more of an ugly way.

Integration Test

Integration Test setting comes default in SBT. It's just a matter of configuring the modules with these settings in the build.sbt

lazy val module1 = (project in file("module1"))
  .configs(IntegrationTest)
.settings(
Defaults.itSettings,
  libraryDependencies += scalatest % "it,test"
)

Adding "it, test" at the end of dependencies enables that lib to be available to both standard unit test & integration test packages. After this, you have a new location to write an integration test.

src/it/scala
src/it/java
src/it/resources

I have created an integration test per module and placed in the above location now to execute the integration test use IntegrationTest / testOnly or it:test in SBT console, and the output as:

[info] Compiling 1 Scala source to /Users/neenad/codes/tutorials/scala/sbt-learnings/module1/target/scala-2.11/it-classes ...
[info] Run completed in 9 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
this is a integration test
[info] Module2IntegrationTest1:
[info] - integration test
[info] Run completed in 194 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Done compiling.
module1 integration test
[info] Module1IntegrationTest:
[info] - integration test
[info] Run completed in 134 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2 s, completed 17 Mar, 2019 5:52:42 PM

End-To-End Tests

Like integration test end-to-end configuration is not provided by SBT. We would need to create them and apply to the module. I have created an E2E object in the project folder and, the contents are as below:

import sbt.{Configuration, Defaults, Test, inConfig}
import sbt._
import sbt.Keys._
object E2E {
  final val E2ETest = Configuration.of("EndToEndTest", "e2e") extend (Test)
  final val e2eSettings =
    inConfig(E2ETest)(e2eConfig)
  lazy val e2eConfig =
      Defaults.configSettings ++ Defaults.testTasks ++ Seq(
    scalaSource in E2ETest := baseDirectory.value / "src" / "e2e" / "scala",
    javaSource in E2ETest := baseDirectory.value / "src" / "e2e" / "java",
    resourceDirectory in E2ETest := baseDirectory.value / "src" / "e2e" / "resources",
  )
}

Now apply these settings to your modules and add e2e to the library dependencies as we did with the integration test

lazy val module1 = (project in file("module1"))
  .configs(IntegrationTest)
  .configs(E2ETest)
.settings(
  E2E.e2eSettings,
  Defaults.itSettings,
  libraryDependencies += scalatest % "it,test,e2e",
    name := "module1",
  version := "0.1"
)
lazy val module2 = (project in file("module2"))
  .configs(IntegrationTest)
  .configs(E2ETest)
  .settings(
    Defaults.itSettings,
    libraryDependencies += scalatest % "it,test,e2e",
      E2E.e2eSettings,
    name := "module2",
    version := "0.1"
  )
lazy val root = (project in file("."))
  .configs(IntegrationTest)
  .configs(E2ETest)
  .aggregate(module1, module2)
  .enablePlugins(Common)
  .settings(
    E2E.e2eSettings,
    Defaults.itSettings,
    libraryDependencies += scalatest % "it,test,e2e"
  )

we can now create our e2e tests inside the following folders

src/e2e/scala
src/e2e/java
src/e2e/resources

To run the e2e test use EndToEndTest / test or e2e:test in sbt. This command runs only the integration test as shown below:

[info] Run completed in 10 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
this is e2e test
[info] Module1E2ETest:
[info] - module1 e2e test
[info] Run completed in 172 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
module2 e2e test
[info] Module2E2ETest:
[info] - module2 e2e test
[info] Run completed in 250 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 1 s, completed 17 Mar, 2019 5:53:46 PM

Unit Test?

Now, what remains are the unit tests and configuring the unit tests is relatively straightforward. You don't have to do anything at all. 😄 Any test under package src/test/scala gets considered as a unit test and to execute it you have to use your standard test task in sbt, this runs all the unit test in your module:

[info] Compiling 1 Scala source to /Users/neenad/codes/tutorials/scala/sbt-learnings/module1/target/scala-2.11/test-classes ...
[info] Run completed in 9 milliseconds.
[info] Total number of tests run: 0
[info] Suites: completed 0, aborted 0
[info] Tests: succeeded 0, failed 0, canceled 0, ignored 0, pending 0
[info] No tests were executed.
module2 sample unit test
[info] Module2UnitTest:
[info] - sample test
[info] Run completed in 202 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[info] Done compiling.
module1 sample unit test
Hello World
[info] Module1UnitTest:
[info] - sample test
[info] Run completed in 142 milliseconds.
[info] Total number of tests run: 1
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 2 s, completed 17 Mar, 2019 6:06:45 PM

Disabling Parallel Execution:

To disable parallel execution of the e2e test in our code add below line in our E2E settings

E2ETest / parallelExecution := false

Code:

Resources

• SBT Reference

I hope this post helps you to organize your tests such that you get faster feedback by running tests independently and don't have to survive the pain I went through in your project. If you have any doubt, please comment and don't forget to share and subscribe to the newsletter for more posts.

While I was working on this new platform development one of my colleagues told me about Oracle having vagrant images which could provide Oracle 12c to developers working on mac. Then I checked and found that Oracle has created a GitHub repo with the vagrant files both for Linux and Oracle 11 and 12c. After that, I basically took this vagrant files as a base for my work and installed golden gate setup along with confluent package on the Linux box. But you know it was a very tedious task to manually do the installation and all. And if I do a big blog with all the steps you guys would be like?

So, I enhanced the scripts to do the hard work and install Golden Gate + Confluent + enable some configuration for the Oracle database during vagrant box provisioning. And you know what one command does all the setup and configuration for you. Follow the exact given steps in the post and you will be gifted with the magical power of Oracle database, golden gate and confluent. 😂

1. Clone Vagrant Box Images

Clone my oracle vagrant fork from here. Let's wish my pull request get accepted by the oracle contributors team and this work becomes officially a part of Oracle GitHub repository ( finger crossed 🤞🏼).

2. Download Oracle Database Zip

Download the Oracle database 12c zip file from oracle tech network here or here. Use the Oracle Database 12c Release 2 - Linux x86-64 version and download the single file zip which is around 3.2GB.

3. Download Oracle GG and Oracle GG BD

Now, we need the Oracle Golden Gate and Golden Gate for Big Data zip files. These files can be downloaded from here. Download Oracle GoldenGate for Big Data 12.3.1.1.1 on Linux x86-64 & Oracle GoldenGate 12.3.0.1.4 for Oracle on Linux x86-64.

Keep all the downloaded files in the same vagrant git checkout directory.

4. Start Vagrant

At this point, all the pre-requisites are done. Now run vagrant up and this will start installing Oracle database 12c, oracle golden gate, oracle golden gate for big data and confluent oss package on the Linux box.

Wait till everything finishes. At the very end, the setup will print the password for Oracle database make sure you copy that as this will help you to progress with the next steps. Now, you have your CDC dev environment ready on Mac or on any machine having vagrant. 😎

If you are new to Oracle golden gate and want to know about configuring the golden gate for Oracle database follow the below steps in the post.

5. Configure Golden Gate On Vagrant Box

There are two things we need to configure, the first is oracle golden gate and then oracle golden gate for big data. Let's begin by first configuring the golden gate for Oracle database. Please follow the exact step to get it running. You can find more information about the following command on oracle golden gate documentation.

1. Load Sample Schema To Oracle Database.

We need a schema present in the installed oracle database so that we can configure the golden gate to listen to the changes happening on this schema. Well, Oracle setup comes with HR schema SQL files present in the installation. We will load the same into our database.

use vagrant ssh command to ssh into the virtual machine

Log in to the database using SQL shell as a sysdba:

sqlplus / as sysdba

Alter the session to allow Oracle scripts to execute:

alter session set "_ORACLE_SCRIPT"=true;

Alter the session to use the PDB database (inside SQL shell):

alter session set container=ORCLPDB1;

Load the schema and for all the input asked by the below command provide proper responses:

@?/demo/schema/human_resources/hr_main.sql

Execute the below commands as an oracle user, switch the vagrant user to Oracle using

sudo su - oracle

2. Configure Oracle Golden Gate

1. Go to Oracle golden gate installation directory. cd /u01/ogg

2. Open golden gate console. ./ggsci

3. Start manager. > start mgr

4. Log in to the database. > DBLOGIN USERID SYSTEM@localhost:1521/ORCLCDB PASSWORD [password copied while installation]

5. Register Extract. > REGISTER EXTRACT EXT1 DATABASE CONTAINER (ORCLPDB1)

6. Enable schema-level supplemental logging for a table. > ADD SCHEMATRANDATA ORCLPDB1.HR ALLCOLS

7. Create an Extract group. > ADD EXTRACT EXT1, INTEGRATED TRANLOG, BEGIN NOW

8. Create a trail for online processing on the local system and Associate it with an Extract group. > ADD EXTTRAIL ./dirdat/lt EXTRACT EXT1

9. Create EXT1 parameter file and paste the content in the file. > EDIT PARAM EXT1

EXTRACT EXT1
USERID SYSTEM@ORCLCDB, PASSWORD [password copied during installation]
EXTTRAIL ./dirdat/lt
SOURCECATALOG ORCLPDB1
TABLE HR.*;

1. Start Extract EXT1. > start ext1

2. View status of manager and ext1. > info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING
EXTRACT     RUNNING     EXT1        00:00:00      00:00:00

The Oracle golden gate for Oracle 12c is configured. Now, its time to configure Oracle golden gate for big data which will push all the changes/ CDC events to confluent Kafka.

3. Configure Oracle Golden Gate For Big Data

Before following the below steps make sure you add java path to $LD_LIBRARY_PATH environment variable in your .bashrc file. The location of the java is /usr/lib/jvm/java-1.8.0-openjdk-XXXXX/jre/lib/amd64/server. Replace XXXXX with your current version installed on during the installation.

1. Change directory to oggbd cd /u01/oggbd

2. Open golden gate for bd console. ./ggsci

3. Create default directories. > CREATE SUBDIRS After doing this exit the console using exit command, and create the following files in dirprm directory.

• kafkaconnect.properties

• rkafka_handler.props

• rkafka.prm

1. Login to GG console again using step 2 then execute the below command and add the contents to the param file > edit param mgr

    PORT 7801
   

2. Start the manager. > start mgr

3. Create replicat group rkafka. > add replicat rkafka, exttrail /u01/ogg/dirdat/lt

4. Start the replicat. > start rkafka

5. View status of all if everything is properly configured. > info all

    Program     Status      Group       Lag at Chkpt  Time Since Chkpt
    MANAGER     RUNNING
    REPLICAT    RUNNING     RKAFKA      00:00:00      00:00:09
   

4. Verify Setup

1. Login to the database and add row to any table like:

    > sqlplus / as sysdba
    SQL> alter session set container=ORCLPDB1;
    SQL> INSERT INTO HR.REGIONS(REGION_ID, REGION_NAME) VALUES(47, 'FOO');
    SQL> COMMIT;
   

2. Check if the kafka topics are created for the data inserted tables > kafka-topics --zookeeper localhost:2181 --list

    __confluent.support.metrics
    __consumer_offsets
    _confluent-ksql-default__command_topic
    _schemas
    connect-configs
    connect-offsets
    connect-statuses
    ora-ogg-HR-REGIONS-avro
   

3. To check the data in the kafka topics run - > kafka-avro-console-consumer --bootstrap-server localhost:9092 --topic ora-ogg-HR-REGIONS-avro --from-beginning

You can run both kafka-topics and kafka-avro-console-consumer from the host machine where you are running vagrant. This will help you while coding to connect to Kafka instance present in the vagrant vm. Also, The Oracle database is running on port 1521 to which you can connect from your java code or from your IDE.

This vagrant images is really helpful to me and hope it will also help you guys while developing your CDC related applications. For any issues, during setup or configuring please do comment on the post.

Other Information:

• Zookeeper Port: 2181

• Kafka Broker: 9092

• Confluent Schema Registry: 8081

• Oracle DB Port: 1521

Don't forget to share the post with your friends and colleagues.

References:

1. Oracle Repository

2. Own Personal Project Repository

I am writing this post because I have to search a lot of things to get it working and there is no single place to get the exact information on how to run a Spark job on a local Kubernetes cluster. I thought writing a post on my findings will enable others not to go through the same pain as I did.

In this post, we will take a look at how to set up a Kubernetes cluster using minikube on a local machine and how to run an Apache spark job on top of it. We are not going to write any code for this, a few examples are already available within the Apache Spark distribution and we are going to use the same example jar and run the SparkPi program from it onto our Kubernetes cluster.

Prerequisites

Before we start please make sure you have the following prerequisites installed on your local machine.

I am running the tools on Mac High Sierra.

• Apache Spark (v2.3.0)

• Kubernetes (v.1.9.3)

• Docker

• Minikube

• VirtualBox

Start Minikube

We can start minikube locally by simply running the below command.

> minikube start
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

Minikube is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes cluster inside a VM on a local machine. It enables us to try out Kubernetes or develop it day-to-day.

When minikube starts it starts with a single-node configuration and takes 1Gb of Memory and 2 cores of CPU by default, however, for running spark this requirement will not suffice and the jobs will fail (and they do as I have tried multiple times) so we have to increase both Memory and CPU cores for our minikube cluster.

There are three different ways to do this. However, for me not all worked and that’s how I came to know about the different ways to change the minikube configuration. You can use any one of the below ways to change the configuration.

Pass Configuration To Minikube Start Command

You can directly pass the memory and CPU options to the minikube start command like:

> minikube start --memory 8192 --cpus 4
Starting local Kubernetes v1.9.0 cluster...
Starting VM...
Getting VM IP address...
Moving files into cluster...
Setting up certs...
Connecting to cluster...
Setting up kubeconfig...
Starting cluster components...
Kubectl is now configured to use the cluster.
Loading cached images from config file.

This will start a minikube cluster with 8Gb of memory and 4 cores of CPU. (somehow for me this didn't work)

Modify minikube config

We can change the minikube config with the config command in minikube cli. The config command allows setting different config options for minikube like memory, CPUs, vm-driver, disk-size etc. To see all the available options use the> minikube config command it will list all the available options that can be modified.

> minikube config set memory 8192

## These changes will take effect upon a minikube delete and then a minikube start

> minikube config set cpus 4

## These changes will take effect upon a minikube delete and then a minikube start

After setting the configuration using the config command we need to delete our previous running cluster and start a new one. To delete minikube cluster run > minikube delete and rerun the start minikube command.

Change Configuration Using VirtualBox

For me, the above two options didn't work and if you are like me you can use this last option which worked for me and hope it works for you. Open the VirtualBox app on your machine and select your VM like the one shown in the below image. RightClick -> Settings on your VM, this will open the configuration page for the Minikube VM.

note: To change configuration using VirtualBox first you need to shutdown the VM if it is already running.

Inside option System -> Motherboard, you can change the memory of the VM using the slider, in my case I have given it 8192MB of memory.

To change the CPU config, go to Processor tab and change it to 4. You can try other options also for me 4 works just fine. Don’t change it to less than 2 otherwise things won’t work. :p

After the configuration changes are done, start the VM using the VirtualBox app or using the above given minikube start command.

Your cluster is running and now you need to have a docker image for spark. Let's see how to build the image next.

Creating Docker Image For Spark

Make sure you have Docker installed on your machine and the spark distribution is extracted.

Go inside your extracted spark folder and run the below command to create a spark docker image

some of the output logs are excluded.

> ./bin/docker-image-tool.sh -m -t spark-docker build
./bin/docker-image-tool.sh: line 125: ================================================================================: command not found
Sending build context to Docker daemon  256.5MB
Step 1/16 : FROM openjdk:8-alpine
 ---> 224765a6bdbe
Step 2/16 : ARG spark_jars=jars
 ---> Using cache
 ---> dd42fdc28d7a
Step 3/16 : ARG img_path=kubernetes/dockerfiles
 ---> Using cache
 ---> 570fc343f883
Step 4/16 : RUN set -ex &&     apk upgrade --no-cache &&     apk add --no-cache bash tini libc6-compat &&     mkdir -p /opt/spark &&     mkdir -p /opt/spark/work-dir     touch /opt/spark/RELEASE &&     rm /bin/sh &&     ln -sv /bin/bash /bin/sh &&     chgrp root /etc/passwd && chmod ug+rw /etc/passwd
 ---> Using cache
 ---> 705ae89cb075
Step 5/16 : COPY ${spark_jars} /opt/spark/jars
 ---> Using cache
 ---> 506ab8c02abf
Step 9/16 : COPY ${img_path}/spark/entrypoint.sh /opt/
 ---> Using cache
 ---> 03959bfa5250
Step 10/16 : COPY examples /opt/spark/examples
 ---> Using cache
 ---> 5a2f91a7ce3e
Step 11/16 : COPY data /opt/spark/data
 ---> Using cache
 ---> 58090cef2be4
Step 13/16 : RUN echo $(ls /opt/spark/examples/jars/)
 ---> Using cache
 ---> 0a6c27628ac7
Step 14/16 : ENV SPARK_HOME /opt/spark
 ---> Using cache
 ---> 7f32a22e0196
Step 16/16 : ENTRYPOINT [ "/opt/entrypoint.sh" ]
 ---> Using cache
 ---> 01daf3302719
Successfully built 01daf3302719
Successfully tagged spark:spark-docker

> docker image ls
REPOSITORY                                                TAG                 IMAGE ID            CREATED             SIZE
spark-docker                                              v0.1                01daf3302719        2 days ago          347MB

Now if you run > docker image ls you will see the docker build available on your local machine. Make a note of this image name we need to provide the image name to spark-submit command.

There is a push option available to the above command which enables you to push the docker image to your own repository this, in turn, will enable your production kubernetes to pull the docker image from the configured Docker repository. Run the same command without any options to see its usage.

It might happen that the command will not work and will give an error like:

./bin/docker-image-tool.sh: line 125: ================================================================================: command not found
“docker build” requires exactly 1 argument.
See ‘docker build — help’.
Usage: docker build [OPTIONS] PATH | URL | — [flags]
Build an image from a Dockerfile

This is because of an issue with the docker-image-tool.sh file. I have raised a bug for this in Apache Spark JIRA you can see it here.

The issue is under fix but for you to continue with this post what you can do is open the docker-image-tool.sh file present inside the bin folder and after line no 59 add BUILD_ARGS=(), save the file and run the command once again and it will work.

For those who the above command worked without the workaround the issue might have been fixed at the time you are reading this post and you don't have to do anything hurrah!!!

Submit Spark Job

Now, let's submit our SparkPi job to the cluster. Our cluster is ready and we have the docker image. Run the below command to submit the spark job on a kubernetes cluster. The spark-submit script takes care of setting up the classpath with Spark and its dependencies and can support different cluster managers and deploy modes that Spark supports

> spark-submit \                                      
--master k8s://https://192.168.99.100:8443 \
--deploy-mode cluster \
--name spark-pi \
--class org.apache.spark.examples.SparkPi \
--conf spark.executor.instances=3 \
--conf spark.kubernetes.container.image=spark-docker \
local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar

options used to run on Kubernetes are:

• --class: The entry point for your application (e.g. org.apache.spark.examples.SparkPi)

• --master: The master URL for the Kubernetes cluster (e.g. k8s://https://192.168.99.100:8443)

• --deploy-mode: Whether to deploy your driver on the worker nodes (cluster) or locally as an external client (client) (default: client)

• --conf spark.executor.instances=3: configuration property to specify how many executor instances to use while running the spark job.

• --conf spark.kubernetes.container.image=spark-docker: Configuration property to specify which docker image to use, here provide the same docker name from docker image ls command.

• local:///opt/spark/examples/jars/spark-examples_2.11-2.3.0.jar: Path to a bundled jar including your application and all dependencies. The URL must be globally visible inside of your cluster, for instance, an hdfs:// path or a local:// path that is present on all nodes

The package jar should be available cluster-wide either through HDFS, or HTTP or should be available within every packaged docker image so that it will be available to all the executor nodes in our spark program. The local:// address means that the jar is available as a local file on every initialized pod by the spark driver and no IO has to be made to pull the jar from anywhere, this works well when the application jar is large and is pushed to every worker node or shared using any shared filesystem.

Luckily, our spark docker image file packages the example jar in the docker container so we can use it. However, how to package our own application code and push it either on HDFS or as a separate docker image I will write as a separate post.

To check if the pods are started and the spark job is running, open the Kubernetes dashboard available within minikube.

View Minikube Dashboard & Kubernetes Logs

To check the status of our submitted job we can use either the Kubernetes dashboard or view Kubernetes logs. Minikube comes with the dashboard available as an add-on and can be started using the following command.

> minikube dashboard
Opening kubernetes dashboard in default browser...

OR

> minikube service list
|-------------|------------------------------------------------------|--------------------------------|
|  NAMESPACE  |                         NAME                         |              URL               |
|-------------|------------------------------------------------------|--------------------------------|
| default     | kubernetes                                           | No node port                   |
| default     | neo4j                                                | No node port                   |
| default     | neo4j-public                                         | http://192.168.99.100:30388    |
|             |                                                      | http://192.168.99.100:30598    |
| default     | spark-pi-709e1c1b19813e7cbc1aeff45200c64e-driver-svc | No node port                   |
| kube-system | kube-dns                                             | No node port                   |
| kube-system | kubernetes-dashboard                                 | http://192.168.99.100:30000    |
|-------------|------------------------------------------------------|--------------------------------|

Navigate to the URL given by the above command to view the dashboard. The dashboard provides lots of information about cluster memory usage, CPU usage, pods, services, replica set etc. We can also view service logs directly through the dashboard. However, if you don't want to go to the dashboard you can view the Spark Driver log using the> kubectl logs <pod name> command:

> kubectl logs spark-pi-709e1c1b19813e7cbc1aeff45200c64e-driver 
.....
2018-03-07 13:10:35 INFO  TaskSchedulerImpl:54 - Removed TaskSet 0.0, whose tasks have all completed, from pool 
2018-03-07 13:10:35 INFO  DAGScheduler:54 - Job 0 finished: reduce at SparkPi.scala:38, took 1.091617 s
Pi is roughly 3.1389956949784747
2018-03-07 13:10:35 INFO  AbstractConnector:318 - Stopped Spark@53e211ee{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
2018-03-07 13:10:35 INFO  SparkUI:54 - Stopped Spark web UI at http://spark-pi-709e1c1b19813e7cbc1aeff45200c64e-driver-svc.default.svc:4040
2018-03-07 13:10:35 INFO  KubernetesClusterSchedulerBackend:54 - Shutting down all executors
2018-03-07 13:10:35 INFO  KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint:54 - Asking each executor to shut down
2018-03-07 13:10:35 INFO  KubernetesClusterSchedulerBackend:54 - Closing kubernetes client
2018-03-07 13:10:35 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
2018-03-07 13:10:35 INFO  MemoryStore:54 - MemoryStore cleared
2018-03-07 13:10:35 INFO  BlockManager:54 - BlockManager stopped
2018-03-07 13:10:35 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
2018-03-07 13:10:35 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
2018-03-07 13:10:35 INFO  SparkContext:54 - Successfully stopped SparkContext
2018-03-07 13:10:35 INFO  ShutdownHookManager:54 - Shutdown hook called
......

If you see in the logs the program has calculated the Pi value and the container is stopped in the end.

Shutdown Cluster

Shutting down the cluster is very easy, use > minikube stop and it will stop the cluster.

Hope this helps you try running apache spark on the local Kubernetes cluster. Please do comment if you didn’t understand any steps or get any errors while following the steps and I will try to add more details to the post. Also, please do let me know if you liked it and help others by sharing.

Happy Coding...

References

1. Apache Spark Documentation

2. Minikube Github