AssertionError after db reconnection

[etam]

We have:

• a 3 node mariadb 10.6 galera cluster, that uses replication to follow another galera cluster.
• a haproxy node
• an application that connects to the database through haproxy and reads the replication log.

The scenario:

• An update to database config causes the db nodes to be restarted one by one.
• This obviously causes the application to lose connection and reconnect. Because it goes through haproxy it reconnects to a different db node.
• After the reconnection, the AssertionError is raised:

AssertionError: Result length not requested length:
Expected=125.  Actual=51.  Position: 89.  Data Length: 140
[...]
  File "pymysqlreplication/row_event.py", line 565, in rows
    self._fetch_rows()
  File "pymysqlreplication/row_event.py", line 560, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "pymysqlreplication/row_event.py", line 621, in _fetch_one_row
    row["values"] = self._read_column_data(self.columns_present_bitmap)
  File "pymysqlreplication/row_event.py", line 148, in _read_column_data
    values[name] = self.__read_values_name(
  File "pymysqlreplication/row_event.py", line 216, in __read_values_name
    else self.__read_string(1, column)
  File "pymysqlreplication/row_event.py", line 322, in __read_string
    string = self.packet.read_length_coded_pascal_string(size)
  File "pymysqlreplication/packet.py", line 243, in read_length_coded_pascal_string
    return self.read(length)
  File "pymysqlreplication/packet.py", line 144, in read
    return self.packet.read(size)
  File "pymysql/protocol.py", line 73, in read
    raise AssertionError(error)

I have 12 samples right now. There are different values in "Expected=???. Actual=???. Position: ???. Data Length: ???" and in few cases instead of

  File "pymysqlreplication/row_event.py", line 216, in __read_values_name
    else self.__read_string(1, column)

there is

  File "pymysqlreplication/row_event.py", line 223, in __read_values_name
    return self.__read_string(column.length_size, column)

This might be a duplicate of #610 .

Unfortunately for now I don't have more data and I won't be restarting production db nodes just for this.

[sean-k1]

Can you let me know pymyrepli version
@etam

[etam]

The latest 1.0.9

[sean-k1]

@etam I'm guessing that your application is generating binlog rotate when the master node changes, can you tell me how the binlog is written when the master node changes?

[etam]

Can you expand the part "how the binlog is written"? I don't know how to answer that.

[sean-k1]

@etam Can you open the binlog file when the application is disconnected? You can type this command
show binlog events;

[etam]

As I wrote

Unfortunately for now I don't have more data

When I was investigating closer what happened (and it was unfortunately few days after), the binlog events from that time were already gone.

and I won't be restarting production db nodes just for this.

I can try to reproduce it in a test environment, but it might be hard.