Skip to content

AssertionError after db reconnection #631

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
etam opened this issue Mar 13, 2025 · 6 comments
Open

AssertionError after db reconnection #631

etam opened this issue Mar 13, 2025 · 6 comments

Comments

@etam
Copy link
Contributor

etam commented Mar 13, 2025

We have:

  • a 3 node mariadb 10.6 galera cluster, that uses replication to follow another galera cluster.
  • a haproxy node
  • an application that connects to the database through haproxy and reads the replication log.

The scenario:

  • An update to database config causes the db nodes to be restarted one by one.
  • This obviously causes the application to lose connection and reconnect. Because it goes through haproxy it reconnects to a different db node.
  • After the reconnection, the AssertionError is raised:
AssertionError: Result length not requested length:
Expected=125.  Actual=51.  Position: 89.  Data Length: 140
[...]
  File "pymysqlreplication/row_event.py", line 565, in rows
    self._fetch_rows()
  File "pymysqlreplication/row_event.py", line 560, in _fetch_rows
    self.__rows.append(self._fetch_one_row())
  File "pymysqlreplication/row_event.py", line 621, in _fetch_one_row
    row["values"] = self._read_column_data(self.columns_present_bitmap)
  File "pymysqlreplication/row_event.py", line 148, in _read_column_data
    values[name] = self.__read_values_name(
  File "pymysqlreplication/row_event.py", line 216, in __read_values_name
    else self.__read_string(1, column)
  File "pymysqlreplication/row_event.py", line 322, in __read_string
    string = self.packet.read_length_coded_pascal_string(size)
  File "pymysqlreplication/packet.py", line 243, in read_length_coded_pascal_string
    return self.read(length)
  File "pymysqlreplication/packet.py", line 144, in read
    return self.packet.read(size)
  File "pymysql/protocol.py", line 73, in read
    raise AssertionError(error)

I have 12 samples right now. There are different values in "Expected=???. Actual=???. Position: ???. Data Length: ???" and in few cases instead of

  File "pymysqlreplication/row_event.py", line 216, in __read_values_name
    else self.__read_string(1, column)

there is

  File "pymysqlreplication/row_event.py", line 223, in __read_values_name
    return self.__read_string(column.length_size, column)

This might be a duplicate of #610 .

Unfortunately for now I don't have more data and I won't be restarting production db nodes just for this.

@sean-k1
Copy link
Collaborator

sean-k1 commented Mar 14, 2025

Can you let me know pymyrepli version
@etam

@etam
Copy link
Contributor Author

etam commented Mar 17, 2025

The latest 1.0.9

@sean-k1
Copy link
Collaborator

sean-k1 commented Mar 19, 2025

@etam I'm guessing that your application is generating binlog rotate when the master node changes, can you tell me how the binlog is written when the master node changes?

@etam
Copy link
Contributor Author

etam commented Mar 19, 2025

Can you expand the part "how the binlog is written"? I don't know how to answer that.

@sean-k1
Copy link
Collaborator

sean-k1 commented Mar 19, 2025

@etam Can you open the binlog file when the application is disconnected? You can type this command
show binlog events;

@etam
Copy link
Contributor Author

etam commented Mar 19, 2025

As I wrote

Unfortunately for now I don't have more data

When I was investigating closer what happened (and it was unfortunately few days after), the binlog events from that time were already gone.

and I won't be restarting production db nodes just for this.

I can try to reproduce it in a test environment, but it might be hard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants