[Test] Run CI against pyspark 3.4

[WeichenXu123]

Run CI against pyspark 3.4

[codecov]

Codecov Report

Patch and project coverage have no change.

Comparison is base (0b0775a) 86.24% compared to head (a8befd5) 86.24%.

❗ Current head a8befd5 differs from pull request most recent head 2e68138. Consider uploading reports for the commit 2e68138 to get more accurate results

Additional details and impacted files

@@           Coverage Diff           @@
##           master     #791   +/-   ##
=======================================
  Coverage   86.24%   86.24%           
=======================================
  Files          84       84           
  Lines        5090     5090           
  Branches      790      790           
=======================================
  Hits         4390     4390           
  Misses        560      560           
  Partials      140      140           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

[WeichenXu123]

Failure tests are all about legacy dataset (dataset saved by old version petastorm) which we don't need to support it.

petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.4.3] FAILED [ 61%]
petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.7.6] PASSED [ 61%]
petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.5.1] FAILED [ 61%]
petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.4.0] FAILED [ 61%]
petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.7.0] FAILED [ 62%]
petastorm/tests/test_reading_legacy_datasets.py::test_reading_legacy_dataset[file:///petastorm/petastorm/tests/data/legacy/0.6.0] FAILED [ 62%]

[selitvin]

Failure tests are all about legacy dataset (dataset saved by old version petastorm) which we don't need to support it.

@WeichenXu123 : can you please delete these tests?