Sign Up

Have an account? Sign In Now

Sign In

Forgot Password?

Don't have account, Sign Up Here

Forgot Password

Lost your password? Please enter your email address. You will receive a link and will create a new password via email.

Have an account? Sign In Now

Sorry, you do not have a permission to ask a question, You must login to ask question.

Forgot Password?

Need An Account, Sign Up Here
Sign InSign Up

ErrorCorner

ErrorCorner Logo ErrorCorner Logo

ErrorCorner Navigation

  • Home
  • Contact Us
  • About Us
Search
Ask A Question

Mobile menu

Close
Ask a Question
  • Home
  • Contact Us
  • About Us
Home/ Questions/Q 398
Next
Answered
Kenil Vasani
Kenil Vasani

Kenil Vasani

  • 646 Questions
  • 567 Answers
  • 77 Best Answers
  • 26 Points
View Profile
  • 3
Kenil Vasani
Asked: December 11, 20202020-12-11T20:34:34+00:00 2020-12-11T20:34:34+00:00In: Python

Pandas: ValueError: cannot convert float NaN to integer

  • 3

I get ValueError: cannot convert float NaN to integer for following:

df = pandas.read_csv('zoom11.csv')
df[['x']] = df[['x']].astype(int)
  • The “x” is obviously a column in the csv file, but I cannot spot any float NaN in the file, and dont get what does it mean by this.
  • When I read the column as String, then it has values like -1,0,1,…2000, all look very nice int numbers to me.
  • When I read the column as float, then this can be loaded. Then it shows values as -1.0,0.0 etc, still there are no any NaN-s
  • I tried with error_bad_lines = False and dtype parameter in read_csv to no avail. It just cancels loading with same exception.
  • The file is not small (10+ M rows), so cannot inspect it manually, when I extract a small header part, then there is no error, but it happens with full file. So it is something in the file, but cannot detect what.
  • Logically the csv should not have missing values, but even if there is some garbage then I would be ok to skip the rows. Or at least identify them, but I do not see way to scan through file and report conversion errors.

Update: Using the hints in comments/answers I got my data clean with this:

# x contained NaN
df = df[~df['x'].isnull()]

# Y contained some other garbage, so null check was not enough
df = df[df['y'].str.isnumeric()]

# final conversion now worked
df[['x']] = df[['x']].astype(int)
df[['y']] = df[['y']].astype(int)
csvpandaspython
  • 1 1 Answer
  • 9 Views
  • 0 Followers
  • 0
Answer
Share
  • Facebook

    1 Answer

    • Voted
    1. Rohit Patel

      Rohit Patel

      • 0 Questions
      • 98 Answers
      • 0 Best Answers
      • 0 Points
      View Profile
      Best Answer
      Rohit Patel
      2020-12-11T20:34:06+00:00Added an answer on December 11, 2020 at 8:34 pm

      For identifying NaN values use boolean indexing:

      print(df[df['x'].isnull()])
      

      Then for removing all non-numeric values use to_numeric with parameter errors='coerce' – to replace non-numeric values to NaNs:

      df['x'] = pd.to_numeric(df['x'], errors='coerce')
      

      And for remove all rows with NaNs in column x use dropna:

      df = df.dropna(subset=['x'])
      

      Last convert values to ints:

      df['x'] = df['x'].astype(int)
      
      • 4
      • Share
        Share
        • Share on Facebook
        • Share on Twitter
        • Share on LinkedIn
        • Share on WhatsApp

    You must login to add an answer.

    Forgot Password?

    Sidebar

    Ask A Question
    • Popular
    • Kenil Vasani

      SyntaxError: invalid syntax to repo init in the AOSP code

      • 5 Answers
    • Kenil Vasani

      runtimeError: package fails to pass a sanity check for numpy ...

      • 3 Answers
    • Kenil Vasani

      xlrd.biffh.XLRDError: Excel xlsx file; not supported

      • 3 Answers
    • Kenil Vasani

      Homebrew fails on MacOS Big Sur

      • 3 Answers
    • Kenil Vasani

      SQLSTATE[HY000]: General error: 1835 Malformed communication packet on LARAVEL

      • 2 Answers

    Explore

    • Most Answered
    • Most Visited
    • Most Voted
    • Random

    © 2020-2021 ErrorCorner. All Rights Reserved
    by ErrorCorner.com