Skip to main content

RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping

Image
By

ERROR :

RetryInvocationHandler: Exception while invoking ConsistencyCheckerS3FileSystem.mkdirs over null. Retrying after sleeping for 10000ms.


Root cause :

 Problem occurs because of below issues:
  • Retry logic in spark and hadoop systems.
  • When a process of creating a file on s3 failed, but it already updated in the Dynamodb.
  • Manual deletion of files and directory from S3 console.
  • when the hadoop process restarts the process as the entry is already present in the Dynamodb. It throws the consistent error.



Solution :

Try re-run your spark job by cleaning up the EmrFSMetadata in dynamo db.

Follow the steps to clean-up & Restore the intended specific directory in the S3 bucket….

 Deletes all the objects in the path, emrfs delete uses the hash function to delete the records, so it may delete unwanted entries also, so we are doing the import and sync in the consequent steps

Delete all the metadata

emrfs import s3://<bucket>/path
Sync the data between s3 and the metadata.

emrfs delete   s3://<bucket>/path
Retrieves the metadata for the objects that are physically present in s3 into dynamo db



Hopefully this will resolve the issue.


Labels:

Comments

Post a Comment

Popular posts from this blog

Extent report plugin for cucumber framework

By
Extent Reports  are the most popular  reporting  used with Selenium. ExtentReport API makes our life easy to generate interactive  report  with simple configuartions. It supports almost all Java and .NET test frameworks such as TestNG , JUnit , NUnit etc Here we are discussing about  a plugin which is build on  Extent Report specially for Cucumber. This plugin is used to simple out the implementation of  Extent Report  in  Cucumber Framework .  We are creating a maven project to implement the integration of our plugin with cucumber 1. Create new maven project in any tool eclipse/sts/intellij 2. Open pom.xml and update below entries. Step 1 : Add Cucumber Extent Reporter library to Maven Project Add  cucumber-extentsreport <dependency>      <groupId> com.vimalselvam </groupId>      <artifactId> cucumber-extentsreport </artif...
153 comments
Read more

java: You aren't using a compiler supported by lombok, so lombok will not work and has been disabled.

By
  In order to make projects compile with the existing builds of Lombok processor, as a workaround you can use the flag -Djps.track.ap.dependencies=false which should be added to File | Settings | Build, Execution, Deployment | Compiler | Build process VM options field. This will disable collection of dependencies specified by an annotation processor when Filer methods are called
Post a Comment
Read more

Execution default of goal org.springframework.boot:spring-boot-maven-plugin:1.2.3.RELEASE:repackage failed: Unable to find main class

By
Solutions:  Solution 1 : You needed to change the packaging parameter to jar from pom. Also, the repositories , pluginRepositories , the maven-compiler-plugin and the spring-boot-maven-plugin's version and executions weren't needed. Solution 2:  Try mvn install and see if it works Solution 3: Preview: <properties> <!-- The main class to start by executing java -jar --> <start-class> com.mycorp.starter.HelloWorldApplication </start-class> </properties> Solution 4: Enable the main() method in your Application.java. Configure spring-boot-maven-plugin to specify the class with the main class (Spring should find it anyway if you have one, but good to be explicit): Preview: <plugin> <groupId> org.springframework.boot </groupId> <artifactId> spring-boot-maven-plugin </artifactId> <version> ${spring-boot-version} </version>...
Post a Comment
Read more