Skip to main content

RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping

Image
By

ERROR :

RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping for 10000ms.


Root cause :

 Problem occurs because of below issues:
  • Retry logic in spark and hadoop systems.
  • When a process of creating a file on s3 failed, but it already updated in the Dynamodb.
  • Manual deletion of files and directory from S3 console.
  • when the hadoop process restarts the process as the entry is already present in the Dynamodb. It throws the consistent error.



Solution :

Try re-run your spark job by cleaning up the EmrFSMetadata in dynamo db.

Follow the steps to clean-up & Restore the intended specific directory in the S3 bucket….

 Deletes all the objects in the path, emrfs delete uses the hash function to delete the records, so it may delete unwanted entries also, so we are doing the import and sync in the consequent steps

Delete all the metadata

emrfs import s3://<bucket>/path
Sync the data between s3 and the metadata.

emrfs delete   s3://<bucket>/path
Retrieves the metadata for the objects that are physically present in s3 into dynamo db



Hopefully this will resolve the issue.


Labels:

Comments

Post a Comment

Popular posts from this blog

Extent report plugin for cucumber framework

By
Extent Reports  are the most popular  reporting  used with Selenium. ExtentReport API makes our life easy to generate interactive  report  with simple configuartions. It supports almost all Java and .NET test frameworks such as TestNG , JUnit , NUnit etc Here we are discussing about  a plugin which is build on  Extent Report specially for Cucumber. This plugin is used to simple out the implementation of  Extent Report  in  Cucumber Framework .  We are creating a maven project to implement the integration of our plugin with cucumber 1. Create new maven project in any tool eclipse/sts/intellij 2. Open pom.xml and update below entries. Step 1 : Add Cucumber Extent Reporter library to Maven Project Add  cucumber-extentsreport <dependency>      <groupId> com.vimalselvam </groupId>      <artifactId> cucumber-extentsreport </artif...
149 comments
Read more

java: You aren't using a compiler supported by lombok, so lombok will not work and has been disabled.

By
  In order to make projects compile with the existing builds of Lombok processor, as a workaround you can use the flag -Djps.track.ap.dependencies=false which should be added to File | Settings | Build, Execution, Deployment | Compiler | Build process VM options field. This will disable collection of dependencies specified by an annotation processor when Filer methods are called
Post a Comment
Read more

Logging in AWS lambda

By
Problem : Logs are very important aspect of any microservice or any AWS service and in AWS if your rate of generating Cloudwatch logs is high ,then  it can increase your AWS costs significantly. There should be some limit on the growth of the logs to keep costs under control. Solution : There are multiple ways to manage the logging. Let's begin with some of the best practices and then some framework level changes which can help in reducing the overall logging. Best Practices : 1. Most of the times we are only interested to find error scenarios from the logs so it is important to use log.error while logging error cases. Never use other logging levels for logging errors 2. Never use print statements instead use a proper logging framework to log the statements 2. Be diligent about the log statements and categorize them correctly at different log levels : DEBUG , INFO , ERROR 4. Don't log data unless there is a critical need for same. Eve...
Post a Comment
Read more